Uniquely coded color boards for analyzing images

ABSTRACT

Systems and methods for a color board for use in reagent strip testing are disclosed. One implementation may include a color board surface, a first colored reference element printed on the color board surface, and a second colored reference element printed on the color board surface. The color board may also include a test region on the color board surface configured to receive at least one reagent pad. The color board may also include a unique code, and the code may reflect specific chromatic properties associated with each of the first colored reference element and the second color reference element at a time of printing. The unique code may be machine readable to enable a machine to later normalize a comparison color, for determining chromatic properties of the at least one reagent pad.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of priority of U.S. ProvisionalPatent Application No. 62/787,402, filed on Jan. 2, 2019, U.S.Provisional Patent Application No. 62/812,354, filed on Mar. 1, 2019,U.S. Provisional Patent Application No. 62/812,365, filed on Mar. 1,2019, U.S. Provisional Patent Application No. 62/812,373, filed on filedon Mar. 1, 2019, U.S. Provisional Patent Application No. 62/814,922,filed on Mar. 7, 2019, and U.S. Provisional Patent Application No.62/814,925, filed on Mar. 7, 2019, all of which are incorporated hereinby reference in their entirety.

BACKGROUND I. Technical Field

The present disclosure relates generally to the field of imageprocessing for medical purposes. More specifically, the presentdisclosure relates to systems, methods, and devices for using imageanalysis for evaluating medical conditions.

II. Background Information

Computer vision may be used in medical testing to determine quantitativeand qualitative clinical data. Traditionally, regulatory-approvedclinical devices use dedicated hardware such as pre-calibrated scannersthat operate in well-known and monitored capturing and illuminationconditions, together with classifiers that operate based on thecalibrated images derived by the scanners.

In recent years, smartphones have become personal mobile computers withhigh processing power, wireless Internet access, and high-resolutioncamera capabilities. However, turning a smartphone into aregulatory-approved clinical device is challenging for at least threemain reasons. First, there may be a lack of quality uniformity of thesmartphones' cameras. This can occur, for a number of reasons, includingthe fact that the settings and imaging of each brand and model ofsmartphone may differ from one to the next. Even within a particularmodel, there may be slight variations in acquired images. Second, whenusing smartphones across a host of non-uniformly lit environments, localillumination conditions can lead to varying results. Third, non-medicalprofessionals who operate smartphones may have difficulty followingstrict operation procedures.

The disclosed embodiments are directed to providing new and improvedways for using personal communications devices for medical testing.

SUMMARY

Embodiments consistent with the present disclosure provide systems,methods, and devices for capturing and analyzing images for evaluatingmedical conditions. In one example, consistent with the disclosedembodiments, an exemplary system may receive an image depicting a tissuefeature with multiple segments of differing colors and may use the imageto determine the state of the tissue feature. In a second example,consistent with the disclosed embodiments, an exemplary system mayreceive an image depicting a dipstick having one or more reagent padsand may determine an extent of a chemical reaction on the one or morereagent pads.

Consistent with disclosed embodiments, systems and methods for a colorboard for use in reagent strip testing are disclosed. For example,consistent with one embodiment a disclosed system may include a colorboard surface, a first colored reference element printed on the colorboard surface, and a second colored reference element printed on thecolor board surface. The color board may also include a test region onthe color board surface configured to receive at least one reagent pad.The color board may also include a unique code, and the code may reflectspecific chromatic properties associated with each of the first coloredreference element and the second color reference element at a time ofprinting. The unique code may be machine readable to enable a machine tolater normalize a comparison color, for determining chromatic propertiesof the at least one reagent pad.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processing device and perform any of themethods described herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Thepatent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawings will be provided by the Office upon request and paymentof the necessary fee. In the drawings:

FIG. 1A is a schematic illustration of an exemplary system that usesimage data captured by mobile communications devices for medicaltesting, consistent with the present disclosure.

FIG. 1B is a flowchart of an exemplary process for completing a medicalexamination, consistent with the present disclosure.

FIG. 1C is an exemplary flow diagram illustrating communicationsexchanges between different entities implementing the process of FIG.1B.

FIG. 2 is a block diagram illustrating some of the components of thesystem of FIG. 1A, consistent with the present disclosure.

FIG. 3 is a schematic illustration of how two different mobilecommunications devices can obtain the same test results, consistent withthe present disclosure.

FIG. 4A is an illustration of one aspect of the disclosure where theexamined object is a tissue feature.

FIG. 4B is an illustration of another aspect of the disclosure where theexamined object is a dipstick.

FIG. 5A depicts three images of a same reagent captured in differentillumination conditions.

FIG. 5B is an illustration of a color mapping chart associated with thereagent shown in FIG. 5A.

FIG. 6 depicts three screenshots illustrating an exemplary graphicaluser interface (GUI) for guiding a user through a medical testingprocess, in accordance with embodiments of the present disclosure.

FIG. 7 is a flowchart of an exemplary process for analyzing visiblechemical reactions, in accordance with some embodiments of the presentdisclosure.

FIG. 8 is a flowchart of an exemplary process for testing visiblechemical reactions of a reagent, in accordance with some embodiments ofthe present disclosure.

FIG. 9 is an illustration of a color board, consistent with the presentdisclosure.

FIG. 10 is a schematic illustration of a color analysis system,consistent with the present disclosure.

FIG. 11 is a flowchart of a process for analyzing colors, consistentwith the present disclosure.

FIG. 12 is an illustration of a urinalysis kit, consistent with thepresent disclosure.

FIG. 13 is a schematic illustration of a urinalysis kit image processingsystem, consistent with the present disclosure.

FIG. 14 is a flowchart of a process for collecting and analyzingurinalysis information, consistent with the present disclosure.

FIGS. 15A and 15B are illustrations of images of wounds and a colorcomparison surface, consistent with the present disclosure.

FIG. 16 is a depiction of an image processing system for analyzing woundimages, consistent with the present disclosure.

FIG. 17 is a flowchart of a process for analyzing an image of a wound,consistent with the present disclosure.

FIG. 18 is a schematic diagram illustrating one aspect of the system forintegrating results of image-based analysis with electronic medicalrecords, in accordance with disclosed embodiments.

FIGS. 19A and 19B are schematic diagrams illustrating variousconfigurations for the system of FIG. 18.

FIG. 20 is a flow chart illustrating a process of integrating results ofimage-based analysis with electronic medical records, in accordance withdisclosed embodiments.

FIG. 21 illustrates a schematic representation of a method for updatingan electronic medical record (EMR) based on patient generated imagedata, consistent with some exemplary aspects of one embodiment of thedisclosure.

FIGS. 22A-22D illustrate an example user interface in accordance withone exemplary aspect of one embodiment of the disclosure.

FIG. 23 illustrates a flowchart in accordance with one exemplary aspectof an embodiment of the disclosure.

FIG. 24 is a schematic illustration of a method of automaticallychanging insurance status according to a first exemplary aspect of thedisclosure.

FIG. 25 is a logic flowchart illustrating one exemplary aspect of thedisclosure for ensuring compliance among home test takers.

FIG. 26 is a flowchart illustrating another exemplary aspect of thedisclosure.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions, or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples, but is inclusive of general principles described herein inaddition to the general principles encompassed by the appended claims.

The present disclosure is directed to systems and methods for processingimages captured by an image sensor. As used herein, the term “imagesensor” refers to any device capable of detecting and converting opticalsignals in the near-infrared, infrared, visible, and ultravioletspectrums into electrical signals. Examples of image sensors may includedigital cameras, phone cameras, semiconductor charge-coupled devices(CCD), active pixel sensors in complementary metal-oxide-semiconductor(CMOS), or N-type metal-oxide-semiconductor (NMOS, Live MOS). Theelectrical signals may be used to generate image data. Consistent withthe present disclosure, the image data may include pixel data streams,digital images, digital video streams, data derived from capturedimages, and data that may be used to construct a 3D image. The imagedata acquired by the image sensor may be transmitted by wired orwireless transmission to a remote server.

Consistent with the present disclosure, the image sensor may be part ofa camera included in a mobile communications device. The term “mobilecommunications device” refers to any portable device with imagecapturing capabilities that can communicate with a remote server over awireless network. Examples of mobile communications devices include,smartphones, tablets, smartwatches, smart glasses, wearable sensors andother wearable devices, wireless communication chipsets, user equipment(UE), personal digital assistants, laptop computers, and any otherportable pieces of communications equipment. It is noted that the terms“handheld mobile communications device,” “handheld mobile device,”“mobile communications device,” and “mobile device” may be usedinterchangeably herein and may refer to any of the variety of deviceslisted above.

Embodiments of the present disclosure further include analyzing imagesto identify a colorized surface in proximity to a medical analysisregion. As used herein, the term “colorized surface” may broadly referto any surface having planar or nonplanar properties. The colorizedsurface may cover or encapsulate at least a portion of a 2D object (suchas a sheet of paper) or at least a portion of a 3D object (such as a boxor a body part). The colorized surface may include a plurality ofreference elements for enabling light and color calibration. In someembodiments, the colorized surface may be printed on a sticker or aplaster (e.g., adhesive bandage), for example, the colorized surfaceillustrated in FIG. 4A. In other embodiments, the colorized surface maybe printed or otherwise presented on a board, cardstock, plastic or anyother medium adapted to serve as a reference. The colorized surface maybe incorporated into the packaging of a test kit, for example. Onenon-limiting example of a colorized surface is illustrated in FIG. 4B.The image correction enabled by the colorized surface may be used toenable a color correction of an image of an object depicted in themedical analysis region. As used herein, the term “medical analysisregion” may be an area on or near the surface distinct from thecolorized portion of the surface used for color correction where anobject for examination may be placed. The medical analysis region may beof uniform color or varied color so long as other portions of thecolorized surface may be used as references for color correction. In apreferred embodiment, the colorized surface may include an un-colorizedor uniformly colorized region demarcated for object placement. Such adistinct region may be larger than the object to be received thereon. Inother embodiments, the medical analysis region may not be demarcated,permitting the user to independently select a location of objectplacement, so long as enough of the colorized surface remains unblockedfor reference purposes during image analysis.

In some embodiments, the examined object is a skin or other tissue oranatomical feature, and the medical analysis region may include any partof the patient's body depicted in the image. In another embodiment, theexamined object may be a dipstick, and the color of the medical analysisregion may be significantly darker or lighter than a majority of thecolorized surface. For example, the medical analysis region maybe atleast 50% darker than the colorized surface. It is noted that the terms“medical analysis region,” “dipstick placement region,” and “testregion,” may be used interchangeably herein to refer to the same area.

Consistent with the present disclosure, the colorized surface may enableprocessing of the image to determine the colors of the examined object,irrespective of local illumination conditions. The term “irrespective oflocal illumination conditions” refers to the output of an image analysisprocess in which the suggested system rectifies the colors of theexamined object to remove at least some effects of local illumination.Effects of local illumination conditions to be removed, may include oneor more of reflections, shades, light temperature (e.g., soft white,cool white, daylight), and any other condition that may impact thedetection of object color. Additionally, the colorized surface may alsoenable processing of the image to determine the colors of the examinedobject, irrespective of specific image capturing effects associated withthe image capturing device. Examples of the different effects associatedwith the image capturing process that may be removed are describedbelow.

In some embodiments, an image correction factor may be generated basedon the determined local illumination conditions and/or image capturingparameters. The image correction factor may be used to remove one ormore local illumination variations and to determine illuminationinvariant colors of the examined object. The image correction factor maybe used to remove image capturing process effects to determine capturingprocess invariant colors of the examined object. In one example, theinvariant colors may be used to determine an extent of a chemicalreaction on a reagent pad. In another example, the illuminationinvariant colors may be used to determine a skin condition, such as acondition of a wound. In yet another example, the invariant colors maybe used to determine a condition of a tissue, such as skin, oral mucosa,nasal mocosa, and so forth. In an additional example, the invariantcolors may be used to determine properties of biological material, suchas a stool sample, a urine sample, a phlegm sample, a blood sample, awax sample, and so forth.

The term “confidence level” refers to any indication, numeric orotherwise, of a level (e.g., within a predetermined range) indicative ofan amount of confidence the system has that the determined colors of theexamined object are the colors of the examined object irrespective oflocal illumination conditions and/or image capturing settings effects.For example, the confidence level may have a value between 1 and 10.Alternatively, the confidence level may be expressed as a percentage orany other numerical or non-numerical indication. In some cases, thesystem may compare the confidence level to a threshold. The term“threshold” as used herein denotes a reference value, a level, a point,or a range of values. In operation, when a confidence level of ameasurement exceeds a threshold (or below it depending on a particularuse case), the system may follow a first course of action and, when theconfidence level is below it (or above it depending on a particular usecase), the system may follow a second course of action. The value of thethreshold may be predetermined for each type of examined object or maybe dynamically selected based on different considerations.

Reference is now made to FIG. 1A, which shows an example of a system 100that uses image analysis to complete a medical examination. System 100may be computer-based and may include computer system components,desktop computers, workstations, tablets, handheld computing devices,memory devices, and/or internal network(s) connecting the components.System 100 may include or be connected to various network computingresources (e.g., servers, routers, switches, network connections,storage devices, etc.) for supporting services provided by system 100.

Consistent with the present disclosure, system 100 may enable user 110to complete a medical examination. In addition, system 100 may enable amedical practitioner 120 to participate in the medical examination usinga mobile communications device 125. The disclosure below that describesthe functionalities of mobile communications device 115 similarlydescribes the functionalities of mobile communications device 125. Inone embodiment, medical practitioner 120 may be a nurse that capturesimages of an object associated with user 110. In another embodiment,medical practitioner 120 may be a physician of user 110 who receives thetest results of the medical examination. In the example illustrated inFIG. 1A, user 110 may use mobile communications device 115 to capture animage 130 that includes a colorized surface 132 and an object to beexamined 134. Image data associated with image 130 may be transmitted toa medical analysis unit 140 for medical testing (directly or via acommunication network). Medical analysis unit 140 may include a server145 coupled to one or more physical or virtual storage devices such as adata structure 146. System 100 may also include or be connected to acommunications network 150 that facilitates communications and dataexchange between different system components and the different entitiesassociated with system 100, such as, healthcare provider 160, insurancecompany 170, and pharmacy 180.

According to embodiments of the present disclosure, medical analysisunit 140 may exchange data with a variety of communication devicesassociated with the different entities associated with system 100. Theterm “communication device” is intended to include all possible types ofdevices capable of exchanging data using communications network 150. Insome examples, the communication device may include a smartphone, atablet, a mobile station, a personal digital assistant, a desktop, alaptop, an IoT device, a dedicated terminal, a server, a cloud, and anyother device that enables data communications. In one implementation,medical analysis unit 140 may receive image data from mobilecommunications device 115, and cause mobile communications device 115 toprovide user 110 with data derived from analysis of examined object 134.In another implementation, medical analysis unit 140 may transmit datato a communications device 165 of healthcare provider 160 for updatingan electronic medical record (EMR) of user 110 stored in data structure166. In another implementation, medical analysis unit 140 may receiveinformation from a communications device 175 of insurance company 170.The received information may identify a group of individuals associatedwith a first insurance status. Thereafter, medical analysis unit 140 mayinitiate medical examinations to determine if there is a likelihood thatthe group of individuals is entitled to a second insurance statusdifferent from the first insurance status. In yet anotherimplementation, medical analysis unit 140 may transmit a medicineprescription to pharmacy 180 for treating user 110 based on the testresult derived from image data captured by mobile communications device115.

Embodiments of the present disclosure may include, access, or otherwiseutilize one or more data structures, such as a database. As uses hereinthe term “data structure” may include any collection of data values andrelationships among them. The data may be stored linearly, horizontally,hierarchically, relationally, non-relationally, uni-dimensionally,multidimensionally, operationally, in an ordered manner, in an unorderedmanner, in an object-oriented manner, in a centralized manner, in adecentralized manner, in a distributed manner, in a custom manner, or inany manner enabling data access. By way of non-limiting examples, datastructures may include an array, an associative array, a linked list, abinary tree, a balanced tree, a heap, a stack, a queue, a set, a hashtable, a record, a tagged union, ER model, and a graph. For example, adata structure may include an XML data structure, an RDBMS datastructure, an SQL data structure or NoSQL alternatives for datastorage/search such as, for example, MongoDB, Redis, Couchbase, DatastaxEnterprise Graph, Elastic Search, Splunk, SoIr, Cassandra, AmazonDynamoDB, Scylla, HBase, SharePoint, Sybase, Oracle and Neo4J. Datastructures, where suitable, may also include document managementsystems. A data structure may be a component of the disclosed system ora remote computing component (e.g., a cloud-based data structure). Datain the data structure may be stored in contiguous or non-contiguousmemory. Moreover, a data structure, as used herein, does not requireinformation to be co-located. It may be distributed across multipleservers, for example, that may be owned or operated by the same ordifferent entities. Thus, the term “data structure” as used herein inthe singular is inclusive of plural data structures.

Consistent with the present disclosure, server 145 may access datastructure 146 to determine, for example, specific chromatic propertiesassociated with colorized surface 132 at the time of printing of thecolorized surface 132. Data structures 146 and data structure 166 mayutilize a volatile or non-volatile, magnetic, semiconductor, tape,optical, removable, non-removable, other type of storage device ortangible or non-transitory computer-readable medium, or any medium ormechanism for storing information. Data structure 146 (and datastructure 166 mutatis mutandis) may be part of server 145 or separatefrom server 145 as shown. When data structure 146 is not part of server145, server 145 may exchange data with data structure 146 via acommunication link. Data structure 146 may include one or more memorydevices that store data and instructions used to perform one or morefeatures of the disclosed embodiments. In one embodiment, data structure146 may include any a plurality of suitable data structures, rangingfrom small data structures hosted on a workstation to large datastructures distributed among data centers. Data structure 146 may alsoinclude any combination of one or more data structures controlled bymemory controller devices (e.g., server(s), etc.) or software.

Consistent with the present disclosure, communications network 150 maybe any type of network (including infrastructure) that supportscommunications, exchanges information, and/or facilitates the exchangeof information between the components of system 100. For example,communications network 150 may include or be part of the Internet, aLocal Area Network, wireless network (e.g., a Wi-Fi/302.11 network), orother suitable connections. In other embodiments, one or more componentsof system 100 may communicate directly through dedicated communicationlinks, such as, for example, a telephone network, an extranet, anintranet, the Internet, satellite communications, off-linecommunications, wireless communications, transponder communications, alocal area network (LAN), a wide area network (WAN), a virtual privatenetwork (VPN), or any other mechanism or combinations of mechanism thatenable data transmission.

The components and arrangements of system 100 shown in FIG. 1A are notintended to be exemplary only and are not intended to limit thedisclosed embodiments, as the system components used to implement thedisclosed processes and features may vary.

FIG. 1B is a flowchart of an exemplary process for completing a medicalexamination according to embodiments of the present disclosure. In someembodiments, the exemplary process is executed by different componentsof system 100. For example, healthcare provider 160, medical analysisunit 140, and user 110. In one embodiment, any action performed byserver 145 may be performed by any combination of mobile communicationsdevice 115, mobile communications device 125, communications device 165,and communications device 175. FIG. 1C illustrates how the exemplaryprocess is implemented by healthcare provider 160, medical analysis unit140, and user's mobile communications device 115.

Example process 190 starts when healthcare provider 160 causes a hometesting kit to be physically provided to user 110 (step 191). Consistentwith the present disclosure, causing the home testing kit to bephysically provided to user 110 may include shipping the test kit touser 110, sending an instruction to a third party to ship a test kit touser 110, physically providing user 110 with a test kit, or conveying atest to user 110 in any other way. For example, shipping instructionsmay be generated, a pick up order may be placed with a shipping company,or the testing kit may be deposited for pickup by a courier. In somecases, healthcare provider 160 may cause home testing kits to bedelivered to a group of individuals identified through information frominsurance company 170. In other cases, healthcare provider 160 may causehome testing kits to be delivered to user 110 in response to a requestfrom medical practitioner 120 or as the result of a request from user110. Alternatively, healthcare provider 160 may automatically cause hometesting kits to be delivered to user 110 based on information about user110 stored in data structure 166. In one example, a physician may havepreviously prescribed annual testing for user 110, or user 110 mighthave met some triggering time-based criteria or health-based criteriathat triggers an indication that user 110 should receive the test kit.In another example, an operator (such as a healthcare provider 160,insurance company 170, etc.) may conduct a query on data structure 166to identify users that meet the selected criteria, and may causedelivery of home testing kits to at least some of the identified users.

Process 190 may continue when user 110 sends a message confirming thereceipt of the home testing kit (step 192). In some embodiments, user110 may send the message directly to healthcare provider 160. In otherembodiments, user 110 may send the message using a dedicated applicationassociated with medical analysis unit 140, and the message may beconveyed to healthcare provider 160. The message may be text or voicebased, or may occur as a button pushed or box checked in response to aprompt on a user interface. Alternatively, the message may simply be thescanning or entry of a code. Thereafter, healthcare provider 160 maysend a verification code to user 110 (step 193). According to oneembodiment, the verification code may be sent in a text message directlyto user 110 after receiving the confirmation message, or may be providedthrough a user interface of an application accessed via a device of user110. As an alternative to an exchange of electronic messages in order toobtaining the verification code, the verification code may be physicallyprovided with the home testing kit in step 191. In such example, step192 and step 193 may be excluded from process 190.

Process 190 may continue when user 110 follows instructions associatedwith the specific medical examination, uses mobile communications device115 to capture image 130, and transmits image data together with (or ina manner that causes it to be associated with) the verification code tomedical analysis unit 140 (step 194). The image data transmitted toimage analysis unit 140 may include image 130, a cropped image withexamined object 134, a processed version of image 130 (e.g., one wherethe color of at least part of the pixels of image 130 was correctedbased on colorized surface 132), or data derived from image 130. In aone aspect of the disclosure, examined object 134 may be a skin feature.According to another aspect of the disclosure, examined object 134 mayinclude a reagent, such as a dipstick with one or more reagent pads.

Process 190 may continue when medical analysis unit 140 determines testresults associated with a state of examined object 134, possibly takinginto account local illumination conditions and/or image capturingsettings effects. In other words, medical analysis unit 140 may inspectthe image of examined object 134 after the effects of the localillumination conditions and/or of the effects of the image capturingsettings are removed. In another example, medical analysis unit 140 mayinspect the image of examined object 134 with a function that takes intoaccount local illumination conditions and/or image capturing settingseffects. When examined object 134 is a dipstick, determining its statemay include determining an extent of a chemical reaction on a least onereagent pad of the dipstick. When examined object 134 is a skin feature,determining the object's state may include determining its condition,for example relative to a previous record of the skin feature. In afirst example, when the skin feature is a wound, medical analysis unit140 may determine from the image data its healing progress. In a secondexample, when the skin feature is a mole, medical analysis unit 140 maydetermine from the image data the likelihood that the mole changed insize or that it has an increased risk of being cancerous. Thereafter,medical analysis unit 140 may transmit the test results to healthcareprovider 160 (step 195), and/or to other entities (such as user 110,medical practitioner 120, insurance company 170, pharmacy 180, and soforth).

Process 190 may continue when healthcare provider 160 initiates anaction based on the received test results. In one embodiment, initiatingan action based on the received test results may include presenting thetest results to medical practitioner 120 (e.g., the user's physician).In another embodiment, initiating an action based on the received testresults may include updating an electronic medic record (EMR) of user110. In another embodiment, initiating an action based on the receivedtest results may include generating a prescription and automatically (orsemi-automatically) forwarding it to pharmacy 180. In anotherembodiment, initiating an action based on the received test results mayinclude sending medical information to user 110 (step 196) or permittingmedical analysis unit 140 to send medical information to user 110. Themedical information transmitted to user 110 may include the testresults, an invitation to schedule an appointment, a prescription, anindication that the user may be eligible for a different insurancecoverage, or any other action that results from the test.

FIG. 1C is a message flow diagram illustrating communications exchangesbetween different entities implementing the process of FIG. 1B. It is tobe understood that the process may be modified consistent withembodiments disclosed herein.

FIG. 2 is an exemplary block diagram of configurations of server 145 andmobile communications device 115. In one embodiment, server 145 andmobile communications device 115 directly or indirectly accesses a bus200 (or other communication mechanism) that interconnects subsystems andcomponents for transferring information within server 145 and/or mobilecommunications device 115. For example, bus 200 may interconnect aprocessing device 202, a memory interface 204, a network interface 206,a peripherals interface 208 connected to I/O system 210, and powersource 209.

Processing device 202, shown in FIG. 2, may include at least oneprocessor configured to execute computer programs, applications,methods, processes, or other software to perform embodiments describedin the present disclosure. For example, the processing device mayinclude one or more integrated circuits, microchips, microcontrollers,microprocessors, all or part of a central processing unit (CPU),graphics processing unit (GPU), digital signal processor (DSP), fieldprogrammable gate array (FPGA), or other circuits suitable for executinginstructions or performing logic operations. The processing device mayinclude at least one processor configured to perform functions of thedisclosed methods such as a microprocessor manufactured by Intel™. Theprocessing device may include a single core or multiple core processorsexecuting parallel processes simultaneously. In one example, theprocessing device may be a single core processor configured with virtualprocessing technologies. The processing device may implement virtualmachine technologies or other technologies to provide the ability toexecute, control, run, manipulate, store, etc., multiple softwareprocesses, applications, programs, etc. In another example, theprocessing device may include a multiple-core processor arrangement(e.g., dual, quad core, etc.) configured to provide parallel processingfunctionalities to allow a device associated with the processing deviceto execute multiple processes simultaneously. It is appreciated thatother types of processor arrangements could be implemented to providethe capabilities disclosed herein.

In some embodiments, processing device 202 may use memory interface 204to access data and a software product stored on a memory device or anon-transitory computer-readable medium. For example, server 145 may usememory interface 204 to access data structure 146. As used herein, anon-transitory computer-readable storage medium refers to any type ofphysical memory on which information or data readable by at least oneprocessor can be stored. Examples include random access memory (RAM),read-only memory (ROM), volatile memory, nonvolatile memory, harddrives, CD ROMs, DVDs, flash drives, disks, any other optical datastorage medium, any physical medium with patterns of holes, a RAM, aPROM, and EPROM, a FLASH-EPROM or any other flash memory, NVRAM, acache, a register, any other memory chip or cartridge, and networkedversions of the same. The terms “memory” and “computer-readable storagemedium” may refer to multiple structures, such as a plurality ofmemories or computer-readable storage mediums located within mobilecommunications device 115, server 145, or at a remote location.Additionally, one or more computer-readable storage mediums can beutilized in implementing a computer-implemented method. The term“computer-readable storage medium” should be understood to includetangible items and exclude carrier waves and transient signals.

Both mobile communications device 115 and server 145 may include networkinterface 206 coupled to bus 200. Network interface 206 may providetwo-way data communications to a network, such as network 150. In FIG.2, the wireless communication between mobile communications device 115and server 145 is represented by a dashed arrow. In one embodiment,network interface 206 may include an integrated services digital network(ISDN) card, cellular modem, satellite modem, or a modem to provide adata communication connection over the Internet. As another example,network interface 206 may include a wireless local area network (WLAN)card. In another embodiment, network interface 206 may include anEthernet port connected to radio frequency receivers and transmittersand/or optical (e.g., infrared) receivers and transmitters. The specificdesign and implementation of network interface 206 may depend on thecommunications network(s) over which mobile communications device 115and server 145 are intended to operate. For example, in someembodiments, mobile communications device 115 may include networkinterface 206 designed to operate over a GSM network, a GPRS network, anEDGE network, a Wi-Fi or WiMAX network, and a Bluetooth® network. In anysuch implementation, network interface 206 may be configured to send andreceive electrical, electromagnetic or optical signals that carrydigital data streams representing various types of information.

Both mobile communications device 115 and server 145 may also includeperipherals interface 208 coupled to bus 200. Peripherals interface 208may be connected to sensors, devices, and subsystems to facilitatemultiple functionalities. In one embodiment, peripherals interface 208may be connected to I/O system 210 configured to receive signals orinput from devices and to provide signals or output to one or moredevices that allow data to be received and/or transmitted by mobilecommunications device 115 and server 145. In one example, I/O system 210may include a touch screen controller 212, audio controller 214, and/orother input controller(s) 216. Touch screen controller 212 may becoupled to a touch screen 218. Touch screen 218 and touch screencontroller 212 may, for example, detect contact, movement or breakthereof using any of a plurality of touch sensitivity technologies,including but not limited to capacitive, resistive, infrared, andsurface acoustic wave technologies as well as other proximity sensorarrays or other elements for determining one or more points of contactwith the touch screen 218. Touch screen 218 may also, for example, beused to implement virtual or soft buttons and/or a keyboard. While atouch screen 218 is shown in FIG. 2, I/O system 210 may include adisplay screen (e.g., CRT or LCD) in place of touch screen 218. Audiocontroller 214 may be coupled to a microphone 220 and a speaker 222 tofacilitate voice-enabled functions, such as voice recognition, voicereplication, digital recording, and telephony functions. The other inputcontroller(s) 216 may be coupled to other input/control devices 224,such as one or more buttons, rocker switches, thumbwheel, infrared port,USB port, and/or a pointer device such as a stylus.

With regard to mobile communications device 115, peripherals interface208 may also be connected to an image sensor 226, a motion sensor 228, alight sensor 230, and/or a proximity sensor 232 to facilitate imagecapturing, orientation, lighting, and proximity functions. Other sensors(not shown) may also be connected to the peripherals interface 208, suchas a temperature sensor, a biometric sensor, or other sensing devices tofacilitate related functionalities. In addition, a GPS receiver may alsobe integrated with, or connected to, mobile communications device 115,such as GPS receivers typically integrated into mobile communicationsdevices. Alternatively, GPS software may permit a mobile communicationsdevice to access AN external GPS receiver (e.g., connecting via a serialport or Bluetooth).

Consistent with the present disclosure, mobile communications device 115may use memory interface 204 to access memory device 234. Memory device234 may include high-speed random-access memory and/or non-volatilememory such as one or more magnetic disk storage devices, one or moreoptical storage devices, and/or flash memory (e.g., NAND, NOR). Memorydevice 234 may store an operating system 236, such as DARWIN, RTXC,LINUX, iOS, UNIX, OSX, WINDOWS, or an embedded operating system such asVxWorks. The operating system 236 may include instructions for handlingbasic system services and for performing hardware-dependent tasks. Insome implementations, the operating system 236 may be a kernel (e.g.,UNIX kernel).

Memory device 234 may also store communication instructions 238 tofacilitate communicating with one or more additional devices, one ormore computers and/or one or more servers. Memory device 234 mayinclude: graphical user interface instructions 240 to facilitate graphicuser interface processing; sensor processing instructions 242 tofacilitate sensor-related processing and functions; phone instructions244 to facilitate phone-related processes and functions; electronicmessaging instructions 246 to facilitate electronic-messaging relatedprocesses and functions; web browsing instructions 248 to facilitate webbrowsing-related processes and functions; media processing instructions250 to facilitate media processing-related processes and functions;GPS/navigation instructions 252 to facilitate GPS and navigation-relatedprocesses and instructions; capturing instructions 254 to facilitateprocesses and functions related to image sensor 226; and/or othersoftware instructions 258 to facilitate other processes and functions.Memory device 234 may also include application specific instructions 260to facilitate a process for guiding user 110 on the steps of the medicaltesting. For example, application specific instructions 260 may causedisplay of a massage indicative of image insufficiency for medicaltesting.

Each of the above identified instructions and applications maycorrespond to a set of instructions for performing one or more functionsdescribed above. These instructions need not be implemented as separatesoftware programs, procedures, or modules. Memory device 234 may includeadditional instructions or fewer instructions. Furthermore, variousfunctions of mobile communications device 115 may be implemented inhardware and/or in software, including in one or more signal processingand/or application specific integrated circuits. For example, mobilecommunications device 115 may execute an image processing algorithm toidentify objects in a received image. In addition, the components andarrangements shown in FIG. 2 are not intended to limit the disclosedembodiments. As will be appreciated by a person skilled in the arthaving the benefit of this disclosure, numerous variations and/ormodifications may be made to the depicted configuration of server 145.For example, not all components may be essential for the operation ofserver 145 in all cases. Any component may be located in any appropriatepart of server 145, and the components may be rearranged into a varietyof configurations while providing the functionality of the disclosedembodiments. For example, some servers may not include all of theelements in I/O system 210.

As mentioned above, one of the challenges of turning a smartphone into aregulatory-approved clinical device is the lack of uniformity of imagecapture capabilities of smartphones. FIG. 3 illustrates twocommunication devices 115 capturing the same object. When a first mobilecommunications device 115A captures examined object 134 in proximity tocolorized surface 132, a first image 130A is acquired. When a secondmobile communications device 115B captures examined object 134 inproximity to colorized surface 132, a second image 130B is acquired.First image 130A may be different from second image 1308 due todifferences between the incorporated image sensors, differences inlighting conditions from different perspectives, and/or differences inimage sensor settings. For example, first image 130A may be differentfrom second image 1308 because first mobile communications device 115Ahas different white balance settings and different color correctionprofiles than second mobile communications device 115B. The whitebalance settings may be associated with how communication devices 115determines the white point for the image and if any tint should beapplied to the other colors. The color correction profile may beassociated with how communication devices 115 process color saturation,black levels, highlights, and the contrast of colors in the image. Inanother example, first image 130A may be different from second image1308 because first mobile communications device 115A has differenthardware (such as image sensor resolution, dimensions, filters, colorfilters, lenses, crop factor, sensitivity, and so forth). In yet anotherexample, first image 130A may be different from second image 130Bbecause first mobile communications device 115A has different cameraconfiguration (such as exposure time, shutter speed, aperture, ISO, andso forth).

Consistent with the present disclosure, each of image 130A and image1306 may undergo an image correction process 300. Image correctionprocess 300 may include one or more steps to remove (or to compensatefor) local illumination effects and image capturing settings effects.The local illumination effects may result from the type of light sourceused to light the object, the distance of the object from the lightsource, a viewing angle of the object, position of the object, ambientlight conditions, flash usage, exposure time, and so forth. The imagecapturing settings effects result from the type of image sensor 226 usedto capture the object, image resolution, frame rate, gain, ISO, shutterspeed, stereo base, lens, focus, zoom, color correction profile, and soforth. In some embodiments of the disclosure, correcting captured image130 may include reversing any of the tone mapping, color enhancement,white balance, and contrast enhancing of image 130. In addition,correcting image 130 may include simulate standard illuminationconditions and reduce shading and specular effects.

Image correction process 300 is enabled through the use of colorizedsurface 132. Specifically, the qualities of one or more color swaths oncolorized surface 132 may be known in advance. To the extent differencesare detected between the actual colors of colorized surface 132 and animage such as image 130A or image 1306, the system may calculate acorrection factor necessary to rectify and such differences, and thenapply that correction factor to the captured image of object 134.

Image correction process 300 may correct each of image 130A and image1306 differently. For example, image correction process 300 may includeincreasing the red color in image 130A and adding brightness to image130B. After images 130A and 130B separately undergo image correctionprocess 300, system 100 may independently determine test results 302from each of image 130A and image 130B. In accordance with the presentdisclosure, even though image 130A may be different from image 1306,test results 302 will be the same because both images captured the sameknown colorized surface 132 whose colorization is known in advance, andwhich may be used as a basis for generating different correction factorsfor the varying differences. In some embodiments, system 100 may correctcaptured image 130A/130B using metadata associated with the mobilecommunications device that captured image 130. In other embodiments,system 100 may correct captured image 130 without using any informationabout the mobile communications device that captured image 130.

FIG. 4A depicts one embodiment where the examined object is a skinfeature 400. Consistent with this aspect, system 100 is configured tomeasure the distribution of colors of skin feature 400 by comparing themto the colors on colorized surface 132. The colors on colorized surface132 may be selected to include an at least some of the expected range ofcolors of the examined object under various illumination and capturingconditions. It may also include a range of colors from which acorrection factor may be generated. As illustrated in FIG. 4A, colorizedsurface 132 may include a plurality of colored reference elements 405and may be attachable onto a skin area next to skin feature 400. Incertain embodiments, colorized surface 132 may have different formsadapted to a medical condition of user 110 or an expected form andcharacteristics of skin feature 400. In addition, colorized surface 132may have different forms adapted to the expected capturing parameters(e.g., to capturing geometry). For example, colorized surface 132 may beround, elongated, curved, have one or more openings therein toaccommodate skin feature 400, etc.

Consistent with the present disclosure, colorized surface 132 may haveone or more colored reference elements 405 used for calibratingillumination and capturing conditions rather than or in addition torelating to colored reference elements 405 associated with the expectedcolors in skin feature 400. When skin feature 400 and colorized surface132 are captured in a single image, system 100 may determine the truecolors of captured skin feature 400 by correcting image 130. In someembodiments, colorized surface 132 may also include one or morepositioning marks 410 that may be used for image processing purposesand/or for positioning colorized surface 132 accurately with respect toskin feature 400. Moreover, positioning marks 410 may provide areference of a known dimension that may be used to estimate a size,orientation, and/or a form of skin feature 400. In certain embodiments,dimensional marks 410 may be used (e.g., by image analysis unit 140) tocorrect captured image 130 with respect to dimensions and forms and toderive an analysis of size and/or form of skin feature 400 and possiblyof other image features. For example, image analysis unit 140 maycompute the color constancy to determine whether two pixels have thesame color in the real world regardless of illumination conditionsand/or camera parameters.

In some embodiments, system 100 may provide two dimensional measurementsof different sections of skin feature 400 associated with a same color,such as size and shape characteristics (symmetry, boundary length etc.).In additional embodiments, system 100 may track skin feature parametersover time by repeatedly capturing the same skin feature over time. Inthis regard, the dimensional mark may assist in determining variationsover time. In one example, skin feature 400 may include scar tissue or arash that may be monitored daily to track healing progress. In anotherexample, skin feature 400 may be captured weekly or even monthly formonitoring potentially cancerous features or developments. Whencollecting such data over a period of time, an additional step may beadded for verifying that the correction of image 130 is consistentacross the time period in which the data was collected. Correcting image130 may further include taking into account illumination conditions andcapturing parameters associated with previously captured images.Additional details on the first aspect of the disclosure are describedin Applicant's U.S. Pat. No. 10,362,984, which is incorporated herein byreference in its entirety.

FIG. 4B provides an example of a colorized surface for use with adipstick 450 having at least one reagent pad 455. The terms “reagentpads” and “colored test reagent pads” may be used interchangeably hereinto refer to the testing areas on a dipstick. As shown, colorized surface132 may include a dipstick placement region 460 and a plurality ofcalibration elements. The calibration elements may have been selected tocorrespond to the type of dipstick 450. Specifically, the calibrationelements may include a plurality of grey elements 465, and a pluralityof colored reference elements 470. Colorized surface 132 may also beprovided with high contrast elements 475 for enabling fast binary largeobject based (BLOB) colorized surface rectification on mobilecommunications device 115 (illustrated, by way of example only, in FIGS.1A, 2, 3, and 6).

Consistent with the present disclosure, colorized surface 132 mayinclude a plurality of grey elements 465 that exhibit various shades ofgray for improved gamma correction. Colorized surface 132 may alsoinclude a plurality of colored reference elements 470 that may beselected to represent at least some of the expected range of colors ofthe examined object under various possible illumination conditions andvarious image processing capabilities of possible image capturingdevices. The colored reference elements 470 may be surrounded by bordersfor minimizing over smoothing of certain colors by some camera models.In addition, to assist image processing, the color of dipstick placementregion 460 may be more than 25%, 50%, or 75% darker than (or brighterthan) the colors of plurality of colored reference elements.Alternatively, the color of dipstick placement region 460 may be morethan 25%, 50%, or 75% visibly different in other ways from the colors ofthe plurality of colored reference elements.

The non-limiting example of colorized surface 132 depicted in FIG. 4Bshows calibration elements 465 and 470 with geometrical shapes thatdiffer from the geometrical shapes of the reagent pads on dipstick 450.The geometrical shapes may be selected to enable differentiation betweenreagent pads 455 and calibration elements 465 and 470 on colorizedsurface 132. Specifically, colorized surface 132 may include a pluralityof cube-like grey elements 465 having three sides, each having adifferent shade of grey; and a plurality of hexagon-shaped coloredreference elements 470 used as reference values for image colorcorrection. On the depicted colorized surface 132, at least two groupsof grey elements (e.g., group of grey elements 465A and group of greyelements 465B) and at least two groups of colored reference elements(e.g., group of colored reference elements 470A and group of coloredreference elements 470B) may be located on opposing sides of dipstickplacement region 460.

Consistent with embodiments of the present disclosure, colorized surface132 may include at least two groups with the same color scheme and thesame shade scheme. The term “the same color scheme” means that thegroups may have a combination of elements with the same one or morecolor families but not necessarily presented in the same order. Hue iswell known in the art and may be defined as the angle of the color whenmapped into a color space (hue ranges from 0-360 degrees), and the term“color family” may depend on a desired level of accuracy, and may referto colors within a hue range of about 4 to 8 degrees, within a hue rangeof about 1 to 2 degrees, within a hue range of about 10 to 20 arcminutes, and so forth. Similarly, the term “the same shade scheme” meansthat the groups may have a combination of elements with the same levelof shade but not necessarily presented in the same order. Colors thatvary by shade have a different level of darkness, but otherwise share asimilar hue and relative chroma. When the shade is defined in a scale of1 to 50, the same level of shade may refer to two elements having ashade value that varies by ±1, by ±0.1, by ±0.01, and so forth,depending on a desired level of accuracy.

As mentioned above, the plurality of colored reference elements 470 oncolorized surface 132 may represent at least some of the expected rangeof colors of the examined object (for example, after going throughdifferent chemical processes, under various possible illuminationconditions, and under various image processing capabilities of differingimage capture devices). On the colorized surface depicted in FIG. 4B,the letters A-H and the numbers 1-10 are added for discussion purposes,to demarcate an 8×10 matrix of colored reference elements 470. Thefollowing table provides example values for the plurality of coloredreference elements 470 in the form of color codes. Each color code is asix-digit code used in HTML, CSS, SVG, and other computing applicationsto represent the specific color used in the corresponding coloredreference element.

A B C D E F G H 1 cfbf6a 9c986f 5c876c 246c6a 246c6a 5c876c 9c986fcfbf6a 2 d5d2ce dbafb7 dd9ea4 d98a98 d98a98 dd9ea4 dbafb7 d5d2ce 3d3cdcd d9b0b9 cea099 d28c84 d28c84 cea099 d9b0b9 d3cdcd 4 bfb99c ae9d9d9a7e8b 7b506d 7b506d 9a7e8b ae9d9d bfb99c 5 d7d473 a59762 5f736c 204d58204d58 5f736c a59762 d7d473 6 d4ca7b c5b172 a8a478 949575 949575 a8a478c5b172 d4ca7b 7 dbd8d5 ddc9bf e0b9af e3ac9a e3ac9a e0b9af ddc9bf dbd8d58 cacac6 c4bcaf b4a7ba 9f8ba0 9f8ba0 b4a7ba c4bcaf cacac6 9 d0976bae7880 7e7e91 536c98 536c98 7e7e91 ae7880 d0976b 10 294e57 62623f a08640dd8d24 dd8d24 a08640 62623f 294e57

In one embodiment, dipstick 450 may include a number of reagent pads455, and colorized surface 132 may include at least the same number ofgroups of colored reference elements with the same color scheme and thesame shade scheme (e.g., multiple groups of differing shades and a samecolor such as groups 470A and 470B). In another embodiment, at least onegroup of colored reference elements may be used in calculating anormalized value (such as a normalized color) for a first reagent padand at least one other group of colored reference elements may be usedin calculating a normalized value (such as a normalized color) for asecond reagent pad. In addition, different groups of colored referenceelements may be used for detecting different reactions in a singlereagent pad or in different reagent pads. In one embodiment, one or moregroups of colored reference elements may be used in calculating anormalized value (such as a normalized color) for both the first reagentpad and the second reagent pad.

In the colorized surface illustrated in FIG. 4B, groups 465A and 465Bserve as a first example of two groups that share the same color schemeand the same shade scheme. Each of groups 465A and 465B has fiveelements of a single color (in this example, gray color), and with allthe elements in each group sharing the same three shades. Groups 470Aand 470B serve as a second example of two groups with the same colorscheme and the same shade scheme. Both groups 470A and 470B have fourelements in differing shades of exactly or substantially a single color(in this example, pink). In this example, group 470B is a mirror imageof group 470A. Specifically, 470A1 has the same level of shade as 470B1,470A2 has the same level of shade as 470B2, 470A3 has the same level ofshade as 470B3, and 470A4 has the same level of shade as 470B4. In someembodiments, colorized surface 132 may include at least two coloredreference elements of differing shades of a same color. The at least twocolored reference elements may be located adjacent each other and on asame side of dipstick placement region 460 (for example, coloredreference elements 470A2 and 470A3); or be located on opposing sides ofdipstick placement region 460 (for example, colored reference elements470A2 and 470B3).

Groups 470C and 470D serve as a third example to two groups sharing thesame color scheme and the same shade scheme. Both groups 470C and 470Dhave four elements with the same four differing colors and exactly orsubstantially a single shade. Specifically, 470C1 is of the same colorfamily as 470D1, 470C2 is of the same color family as 470D2, 470C3 is ofthe same color family as 470D3, and 470C4 is of the same color family as470D4.

The plurality of calibration elements associated with colorized surface132 may have an axis of reflection symmetry in the middle of dipstickplacement region 460. However, other colorized surfaces may havedifferent types of symmetry (e.g., rotational symmetry, or translationalsymmetry), with different symmetry axes, or no symmetry at all.Consistent with the present disclosure, colorized surface 132 mayinclude a plurality of pairs of colored reference elements. Members of apair of colored reference elements may share substantially a same color(for example, same color family, exactly or approximately the samedistribution of color components with different magnitude, etc.) andshare substantially a same shade (i.e., same level of shade). Whencolorized surface 132 has a certain type of symmetry, the members ofeach pair of colored reference elements are substantiallyindistinguishable from each other. In one embodiment, the first and thesecond colored reference elements of a pair of colored referenceelements may be detected in a captured image on opposing sides ofcolorized surface 132. In the depicted example, dipstick placementregion 460 may be positioned between the pairs colored referenceelements. For example, 470A1 and 470B1 may form a first pair of coloredreference elements, 470A2 and 470B2 may form a second pair of coloredreference elements, 470A3 and 470B3 may form a third pair of coloredreference elements, and 470A4 and 470B4 may form a fourth pair ofcolored reference elements.

As shown in FIG. 4B, colorized surface 132 may include at least tenpairs of colored reference elements 470. Alternative embodiments mayinclude more or less pairs. For example, alternative embodiments mayinclude at least twenty pairs of colored reference elements 470, atleast thirty pairs of colored reference elements 470, or even more. Asis described in succeeding paragraphs, the pairs of colored referenceelements may be used to provide a color baseline for comparison with thereagent pads. For example, unless for the relative position of the colorboard with respect to the image sensor, and different illuminationconditions at the locations of the two or more reference elements, twoor more reference elements that share a same color and shade would beexpected by the system to appear identically in a captured image, or maybe expected to have particular appearances based on their relativepositions, given that the positions are known in advance. (i.e.,coordinates of each reference element within the color board are known,for example with respect to other elements of the color board, such asposition markers 474 and 475.) Moreover, if a reference element color isknown in advance as is the case with the colored surface whose colorsmay be precisely controlled during printing, any differences determinedthrough image capture may be corrected by the system in order to ensurethat an accurate reading is achieved. (i.e., a correction factor may beapplied to the reagent pad to correct for differences between expectedcolors and shades of captured reference elements and the colors actuallydetected as the result of image capture). Therefore, when two or morereference elements that share a same color and shade appear different inthe captured image, the difference between them may be attributed to therelative position of the color board with respect to the image sensorand/or to the different illumination conditions at the locations of thetwo or more reference elements. Based on the known coordinates of thetwo or more reference elements, this measured effect may be interpolatedand/or extrapolated to estimate the effect of the relative position ofthe color board with respect to the image sensor and/or of the differentillumination conditions at a selected location, such as a location of areagent pad. Moreover, in some embodiments, the relative position ofdifferent regions (such as a region of a reference element, a region ofa reagent pad, etc.) with respect to the image sensor may be determined(for example using geometrical pose estimation algorithms, using amachine learning model trained to determine the relative position of thecolor board, etc.), and the interpolation and/or extrapolation may befurther based on the determined relative positions of the two or morereference elements and the reagent pad.

Generally, colorized surface 132 may include more colored referenceelements 470 than an expected number of reagent pads 455 on dipstick450. In the illustrated example, dipstick 450 includes ten reagent pads455, and colorized surface 132 includes eighty colored referenceelements 470. Consistent with the present disclosure, the number ofcolored reference elements 470 on colorized surface 132 may be at leasttwo times, four times, six times, or more than the number of reagentpads 455 on dipstick 450. Additional details pertaining to thisdisclosure are described in Applicant's U.S. Pat. No. 10,068,329, whichis incorporated herein by reference in its entirety.

FIG. 5A depicts three images 130 of a dipstick for diagnosing a UrinaryTract Infection (UTI). The UTI dipstick includes three reagent pads, afirst reagent pad 455A for measuring blood in the urine, a secondreagent pad 455B for measuring Nitrite in the urine, and a third reagentpad 455C for measuring Leukocytes in the urine. Each one of the imageswas captured under different illumination conditions. Specifically,image 130A includes a depiction 500 of a UTI dipstick that was capturedunder daylight, image 1306 includes a depiction 505 of a UTI dipstickthat was captured under tungsten bulb light, and image 130C includes adepiction 510 of a UTI dipstick that was captured under fluorescentlight. As shown in the figure, images 130A, 1306, and 130C vary in colordue to the differing lighting conditions. The color boards depicted inFIG. 5A also includes a unique QR code 515 that may reflect specificchromatic properties associated with at least some of the plurality ofcolored reference elements at a time of printing. The unique QR code maybe machine readable to enable a processing device (e.g., server 145) tolater normalize a comparison color, for determining chromatic propertiesof at least one of the examined objects.

As mentioned above, the colorized surface enables processing of theimage to determine the colors of the examined object (e.g., the UTIdipstick), irrespective of local illumination conditions and imagecapturing settings effects. In other words, the system may allow foraccurate diagnostic results, across various lighting conditions, acrossvarious image capture settings, and across various image capturedevices. To determine the colors of the examined object, each coloredreference elements may be associated with a known color (for example,lightness, chroma, saturation, CYMK values, RGB values, and so forth).In one example, the known color may be determined by the type of thecolorized surface. In another example, the known color may be obtainedusing method 1100 and/or using step 1106 as described below. Theappearance of the colored reference element in an image (e.g., image130) may be analyzed to determine the perceived color of the coloredreference element in the image (for example, brightness, colorfulness,saturation, CYMK values, RGB values, and so forth). The known color maybe compared with the perceived color to determine the effects of localillumination conditions and/or the effects of image capturing settingson the perceived color. For example, brightness and colorfulness (or afunction thereof) may be compared with the lightness and chroma (or afunction thereof). Consistent with the present disclosure, thedetermined effects of the local illumination conditions and/or thedetermined effects of the image capturing settings on the perceivedcolor may be measured as a transformation function, as a parameter to atransformation function, as a magnitude of the effect, and so forth. Inone embodiment, the determined effects on the perceived color at two ormore colored reference elements positioned at known locations may beextrapolated and/or interpolated to estimate the effects of the localillumination conditions and/or the effects of the image capturingsettings on a perceived color of a colored reference element positionedat a different location. For example, extrapolation and/or interpolationalgorithms may be used to determine the effects of the localillumination conditions and/or the effects of the image capturingsettings on a perceived color of a colored reference element positionedat a known location. Some non-limiting examples of such extrapolationand/or interpolation algorithms may include linear, polynomial, conic,piecewise constant, spline, and so forth.

Consistent with the present disclosure, a machine learning model may betrained using training examples to estimate the effects of the localillumination conditions and/or the effects of the image capturingsettings on perceived colors of selected colored reference elements fromimages of colored reference elements. The trained machine learning modelmay be used to estimate the effects on a perceived color of a selectedcolored reference element from the appearance of a plurality of coloredreference elements in reference image data. The trained machine learningmodel may also be used for calculating normalized colors of selectedcolored reference elements from reference image data and/or estimate thenormalized color of a selected element from the reference image data. Inone embodiment, the training examples may include image data of aplurality of colored reference elements and a selected colored referenceelement, together with an indication of the effects of the localillumination conditions and/or the effects of the image capturingprocess on a perceived color of the selected colored reference element.In another embodiment, the training examples may include image data of aplurality of colored reference elements and a selected colored referenceelement, together with an indication of the desired normalized color ofthe selected colored reference element.

In one implementation, the estimated effects of the local illuminationconditions and/or the estimated effects of the image capturing settingson a perceived color of a selected color reference element may be usedto reverse the effects to obtain a normalized appearance of the selectedcolor reference element. For example, the estimated effects may be in aform of a function, and an inverse of the function may be applied to theappearance of the selected color reference element to obtain thenormalized appearance of the selected color reference element. Inanother example, the estimated effects may be in a form of a factor, andthe factor may be used to calculate the normalized appearance of theselected color reference element from the appearance of the colorreference selected element.

FIG. 5A also depicts a version of colorized surface 132 different fromthe version of colorized surface 132 depicted in FIG. 4B. Consistentwith the present disclosure, the calibration elements of a colorizedsurface may have been selected to correspond to the type of dipstickbeing examined. Specifically, this version of colorized surface mayinclude calibration elements (i.e., the grey elements and the coloredreference elements) with visual characteristics associated with UTIdipsticks. For example, the number of colored reference elements may beforty not eighty, the color families of colored reference elements maydiffer, the gray elements may be hexagonal, the colored referenceelements may not be surrounded by borders, and any other feature orcolor may be changed so long as the intended function described hereinis achieved. Thus, FIG. 4B is but one example of a colorized surfacethat may to be used to determine an extent of a chemical reaction on aUTI dipstick irrespective of local illumination conditions and/orirrespective of effects of image capturing settings. The exemplaryversion of colorized surface 132 illustrated in FIG. 4B may, asillustrated, include a plurality of colored reference elements ofdiffering shades; the number of colored reference elements may begreater than the number of reagent pads on the examined dipstick; thecolorized surface may include a plurality of groups of colored referenceelements in differing shades of a same color (e.g., 470E and 470F); thecolorized surface may include at least two colored reference elements ofdiffering shades of a same color located adjacent each other and on asame side of a reagent examination region (e.g., 470E1 and 470E2); andthe colorized surface may include at least two colored referenceelements of a same shade of a same color located on an opposing sides ofa reagent examination region (e.g., 470E1 and 470F1). Alternatively,these particular illustrated details may vary across embodiments, solong as the described functionality is achieved.

FIG. 5B depicts color mapping chart 520 next to the reagent test strippads 500, 505, and 510 depicted as they might appear in differinglighting conditions. As mentioned above, UTI dipsticks may include threereagent pads (indicated as 455A, 455B, and 455C on reagent test strip505). In other examples, UTI dipsticks may include less or more reagentpads. In this example, first reagent pad 455A may be used to measureblood in the urine based on the pseudoperoxidase activity of hemoglobin,which catalyzes the reaction of 3,3′,5,5′-Tetramethylbenzidine withbuffered organic hydroperoxide. The resulting color of first reagent pad455A should range from greenish-yellow to greenish-blue and then to darkblue. Second reagent pad 455B may be used to measure Nitrite in theurine based on the reaction of the p-arsanilic acid and nitrite, derivedfrom a dietary nitrate in the presence of bacteria in the urine to forma diazonium compound. The diazonium compound reacts withN-(1-naphthyl)-ethylenediamine in an acidic medium. The resulting colorof second reagent pad 455B should be pink. Third reagent pad 455C may beused to detect Leukocytes (white blood cells) by revealing the presenceof granulocyte esterases. The esterases cleave a derivatized pyrazoleamino acid ester to liberate derivatized hydroxy pyrazole. This pyrazolethen reacts with a diazonium salt. The resulting color of third reagentpad 455C should be purple. In other examples, UTI dipsticks may furtherinclude additional reagent pads, for example for bilirubin, glucose,ketone, pH value, protein, specific gravity, urobilinogen, and so forth.

When reviewing reagent dipsticks 500, 505, and 510 with reference tocolor mapping chart 520 in FIG. 5B, the significance of analyzingresults irrespective of local illumination conditions may be understood.For example, any degree of pink color in second reagent pad 455B isconsidered positive and typically means that the user has a urinarytract bacterial infection. If system 100 was unable to correct fordistortions based on lighting conditions, the colors of second reagentpad 455B, for example would provide an indication of no UTI, when, infact, as reflected on dip stick 500, a UTI exists.

FIG. 6 depicts three screenshots illustrating a graphical user interface(GUI) for guiding user 110 (FIG. 1A) through an exemplary process ofmedical testing. The screenshots are associated with a non-limitingexample of an interactive guidance application and may be displayed onmobile communications device 115. The illustrated interactive guidanceapplication may be designed to appear as a conversation with a chatbot600. In one embodiment, the behavior of chatbot 600 may be defined by adetermined experience level of user 110 (e.g., first time user or areturned user). For example, during a conversation with user 110, aninterpretation engine on the chatbot server (e.g., server 145) maydetermine the experience level of user 110 and navigate the conversationdata structure to fit the user's experience level (e.g., more stepsprovided for first time users). Alternatively, system 100 may determinethe experience level of user 110 by retrieving information from adatabase. Screenshots 602, 604, and 606 illustrate a conversation with auser that received a home testing kit.

Screenshot 602 depicts how the interactive guidance application mayprovide one or more messages indicative of the progress of the testing,by for example, providing messages 608-614. Screenshot 602 also depictshow chatbot 600 may provide user 110 with an indication that the testingof the dipstick is complete, for example through message 614 or message616. Consistent with some embodiments of the present disclosure, thetest results may be automatically transmitted to healthcare provider 160(e.g., as indicated in message 616). User 110 may interact with chatbot600 through an input device (e.g., a keyboard, a touch screen, ormicrophone). For example, interactive guidance application includes a“continue” button 618 for receiving feedback from user 110.

Screenshot 604 depicts how the interactive guidance application mayprovide user 110 data based on the determined extent of the chemicalreaction on the reagent pads of the examined dipstick. For example,interactive guidance application may present to user 110 test results302. In one example, each section in test results 302 may correspondwith a reagent pad on the examined dipstick. In another example, asection in test results 302 may correspond to a result based on two ormore reagent pads. In one embodiment, next to each section, interactiveguidance application may present a “learn more” button 620. Uponselection of the “learn more” button 620, interactive guidanceapplication may provide additional information on the medical meaning ofthe test results. Screenshot 606 depicts chatbot 600 asking user 110 ifhe/she would like to learn how to dispose of the home kit. User 110 mayanswer by selecting the “learn” button 622 or the “skip” button 624. Inaddition, chatbot 600 may provide a notification (not shown) thatincludes an instruction to recapture an image of the dipstick, forexample after determining that the dipstick was improperly placed indipstick placement region 460. Also, in response to a determination thatlocal illumination conditions are insufficient for completing the imageanalysis, chatbot 600 may also instruct user 110 to change at least oneillumination aspect and recapture an image of the dipstick.

FIG. 7 is a flowchart of an example method 700 for analyzing visiblechemical reactions executed by a processing device of system 100,according to embodiments of the present disclosure. The processingdevice of system 100 may include a processor within a mobilecommunications device (e.g., mobile communications devices 115 and 125)or a processor within a server (e.g., server 145) remote from the mobilecommunications device. For purposes of illustration, in the followingdescription reference is made to certain components of system 100. Itwill be appreciated, however, that other implementations are possibleand that any combination of components or devices may be utilized toimplement the exemplary method. It will also be readily appreciated thatthe illustrated method can be altered to modify the order of steps,delete steps, or further include additional steps.

Disclosed embodiments may include receiving from an image sensorassociated with a mobile communications device an image of a reagent padin proximity to a colorized surface having at least one pair of coloredreference elements designed to be used together to determine an extentof a reaction on the reagent pad. As discussed earlier, various types ofimage sensors/mobile communications devices may be used capture variousforms of color references elements adjacent a reagent pad. By way ofexample only, at step 702 in FIG. 7, a processing device (e.g.,processing device 202) may receive from a mobile communications deviceimage data depicting at least one reagent pad in proximity to acolorized surface. The at least one reagent pad (e.g., reagent pad 455)may include a plurality of reagent pads located on a dipstick (e.g.,dipstick 450). The at least one reagent pad may be used for urinalysis,or any other test indicated by a color reagent reaction. The colorizedsurface may have at least one pair of colored reference elementsdesigned to be used together to determine the extent of a reaction onthe at least one reagent pad. Consistent with the present disclosure,the image data includes an image (e.g., image 130) captured by an imagesensor (e.g., image sensor 226) associated with a mobile communicationsdevice (e.g., mobile communications device 115).

Disclosed embodiments may include identifying in an image a reagent pad.This may be achieved by many alternative forms of image analysis, wherethe reagent pad is located within a broader image. For example, at step704, the processing device may identify in the image the at least onereagent pad. In one example, an object detection algorithm may be usedto detect the dipstick in the image, and the reagent pads may beidentified based on their positions on the dipstick. In another example,a machine learning model may be trained using training examples toidentify reagent pads in images, and the trained machine learning modelmay be used to analyze the image and identify the reagent pad. In someembodiments, the processing device may also identify in the image aplurality of position markers (e.g., high contrast elements 475)distinguished from the at least one pair of colored reference elements(e.g., colored reference elements 470). According to an embodiment,among the plurality of position markers, at least one position markerincludes a color that differs from a color of two or more other positionmarkers. In the examples illustrated in FIG. 4B, position marker 474 hasa color scheme that includes a black outer hexagon surrounding a whitehexagon, which itself surrounds a red inner hexagon. In contrast, marker475 replaces the red inner hexagon with a black inner hexagon. Afteridentifying the plurality of position markers, the processing device mayanalyze the image using the plurality of position markers to determinethat an orientation of the at least one reagent pad is an undesiredorientation; for example, the dipstick may be placed upside down ondipstick placement region 460. In response to the determination that theorientation of the at least one reagent pad is at an undesiredorientation, the processing device may provide a notification to a user.The notification may include an instruction to recapture an image of thedipstick and/or a suggestion on how to reposition the orientation of theat least one reagent pad.

Disclosed embodiments may include identifying in the image a firstcolored reference element and identifying in the image a second coloredreference element. This identification may occur through any type ofimage analysis, and the reference elements may be shaped, colored, andlocated according to a designer's preference. In one example, an objectdetection algorithm may be used to detect the color board in the image,and the color reference elements may be identified based on theirpositions on the color board. In another example, pattern recognitionalgorithms may be used to detect known patterns on a color board (suchas position markers 474 and 475), and the color reference elements maybe identified based on their relative positions to the detectedpatterns. In yet another example, a machine learning model may betrained using training examples to identify color reference elements inimages, and the trained machine learning model may be used to analyzethe image and identify the color reference elements. By way of exampleonly, at step 706, the processing device may identify in the image afirst colored reference element and a second colored reference element.In one embodiment, the first and the second colored reference elementsof the at least one pair are arranged in the image on opposing sides ofthe colorized surface.

A colorized surface consistent with one embodiment of the presentdisclosure may include at least ten pairs of colored reference elements.Identifying in the image the first colored reference element and thesecond colored reference element may also include identifying in acaptured image, differences between the first colored reference elementand the second colored reference element. Although the first and thesecond colored reference elements may have been printed with the samecolor and the same shade, local illumination conditions may causedepictions of the first colored reference element and the second coloredreference element to differ in color and/or shade in the captured image.Consistent with the present disclosure, the known colors of the at leastone pair of colored reference elements may be compared with theperceived colors in the captured image, to determine the effects oflocal illumination conditions and/or the effects of image capturingsettings on the at least one reagent. For example, when members of theat least one pair of colored reference elements are arranged on opposingsides of the colorized surface, the system may determine which side iscloser to a light source and use the visual changes between members of apair of colored reference elements to calculate a correction factor forcorrecting the color of the at least reagent pad.

Disclosed embodiments may include using the first colored referenceelement and the second colored reference element to determine an extentof a chemical reaction on the reagent pad irrespective of localillumination conditions. As described earlier, the reference elementsmay be used to correct for local illumination conditions, for therelative position of the colored reference element with respect to theimage sensor, and/or for the image sensor settings. By way of exampleonly, at step 708, the processing device may use the first coloredreference element and the second colored reference element to determinean extent of a chemical reaction on the reagent pad irrespective oflocal illumination conditions, irrespective of the relative position ofthe colored reference element with respect to the image sensor, and/orirrespective of effects of image capturing settings. In one embodiment,the processing device may identify illumination parameters indicative ofthe local illumination conditions based on the at least one pair ofcolored reference elements. Examples of illumination parameters mayinclude the type of illumination source (such as sunlight, LED, neon,incandescent light bulbs, etc.), the frequency of illumination source(for example, of a neon), the location of illumination source, thenumber of illumination sources, and other parameters that may affect thelocal illumination conditions. For example, a machine learning model maybe trained using training example to identify illumination parametersfrom images of color boards, and the trained machine learning model maybe used to analyze an image of a color board to identify theillumination parameters. An example of such training example may includean image of a color board captured under particular illuminationparameters, together with an indication of the illumination parameters.Thereafter, the processing device may determine the extent of thechemical reaction on the reagent pad irrespective of local illuminationconditions based on the identified illumination parameters. In otherwords, by correcting for the local illumination conditions, theprocessing device may determine the extent of reaction irrespective ofthose conditions. For example, a machine learning model may be trainedusing training example to determine the extent of the chemical reactionon reagent pads from images of reagent pads and illumination parameters,and the trained machine learning model may be used to analyze an imageof a reagent pad using identified illumination parameters (for exampleusing the illumination parameters identified as described above) todetermine the extent of the chemical reaction on the reagent padirrespective of local illumination conditions. An example of suchtraining example may include an image of a reagent pad withcorresponding illumination parameters, together with the desired extentof the chemical reaction on the reagent pad to be determined. In anotherembodiment, the processing device may determine image capturingparameters indicative of the effects of the image capturing settings,based on the at least one pair of colored reference elements. In otherwords, since cameras or models of cameras may perceive a scenedifferently based on preset or user-selected settings, the processingdevice may correct for those settings and thereafter determine theextent of a reaction irrespective of those settings.

Additionally, the processing device may calculate a normalized reagenttest color based on the first colored reference element and the secondcolored reference element. Calculating the normalized reagent test colormay include rectifying the colors of the at least one reagent pad toremove effects of local illumination, to remove effects of the relativeposition of the colored reference element with respect to the imagesensor, and/or to remove effects of image capturing settings. In oneexample, a machine learning model may be trained using training examplesto calculate normalized reagent test colors from the color of thereagent pads and colors of reference elements in captured images. Thetrained machine learning model may be used to calculate the normalizedreagent test color from the color of the reagent pad, the color of thefirst colored reference element, and the color of the second coloredreference element in an image. An example of such training example mayinclude a triplet of the color of the reagent pad, the color of thefirst colored reference element, and the color of the second coloredreference element in an image, together with the desired normalizedreagent test color to be determined. When the at least one reagent padincludes a plurality of reagent pads, the processing device may use thenormalized reagent test color associated with a first reagent pad todetermine the normalized reagent test color of a second reagent pad.Thereafter, the processing device may determine the extent of thechemical reaction on the at least one reagent pad irrespective of localillumination conditions based on the normalized reagent test color. Insome cases, when the at least one reagent pad includes a plurality ofreagent pads, the processing device may use one or more normalizedreagent test colors to determine a uniform color for pixels associatedwith each of the depiction of plurality of reagent pads. Thereafter, theprocessing device may determine the extent of the chemical reaction oneach of the plurality of reagent pads based on the uniform colorassociated with each of the plurality of reagent pads.

Disclosed embodiments may include causing the mobile communicationsdevice to provide data based on the extent of the chemical reaction. Thedata which may be provided to either the user and/or transmitted toanother over a network, may include either a test result, an indicatorof a test result, or an instruction related to the extent of a chemicalreaction. By way of example only, at step 710, the processing device maycause the mobile communications device to provide data based on theextent of the chemical reaction. The data provided based on the extentof the chemical reaction may include medical data associated with aplurality of differing urinary properties. For example, the medical datamay include the test results (e.g., test results 302), which may includea conclusion (e.g., positive for UTI) or some indicator from which testresults may be derived (e.g., a value). Alternatively, the data providedbased on the extent of the chemical reaction may include an indicationto contact the user's physician. In one embodiment, causing the mobilecommunications device to provide the data may include causing the mobilecommunications device to transmit the data to a medical entityassociated with a user of the mobile communications device. For example,the medical entity associated with a user may be the user's doctor. Inanother embodiment, causing the mobile communications device to providethe data may include causing the mobile communications device to displaythe data on a screen associated with the mobile communications device.

FIG. 8 is a flowchart of another exemplary method 800 for testingvisible chemical reactions of a reagent pad that may be executed, forexample, by a processing device of system 100, according to embodimentsof the present disclosure. The processing device of system 100 may be aprocessor within a mobile communications device (e.g., mobilecommunications devices 115 and 125) or a processor within a server(e.g., server 145) remote from the mobile communications device. Forpurposes of illustration, in the following description reference is madeto certain components of system 100. It will be appreciated, however,that other implementations are possible and that any combination ofcomponents or devices may be utilized to implement the exemplary method.It will also be readily appreciated that the illustrated method can bealtered to modify the order of steps, delete steps, or further includeadditional steps.

At step 802, a processing device (e.g., processing device 202) mayreceive from a mobile communications device an image of a reagent (e.g.,dipstick 450) with a plurality of colored test reagent pads (e.g.,reagent pad 455) in proximity to a colorized surface (e.g., colorizedsurface 132). The colorized surface may have a plurality of coloredreference elements (e.g., colored reference elements 470) of differingshades. Consistent with the present disclosure, the image (e.g., image130) was captured by an image sensor (e.g., image sensor 226) associatedwith a mobile communications device (e.g., mobile communications devices115).

At step 804, the processing device may use the differing shades of theplurality of colored reference elements to determine local illuminationconditions and/or effects of image capturing settings. As discussedabove with reference to FIG. 4B, the colorized surface may include morecolored reference elements than a number of colored test element pads onthe reagent. In one embodiment, the colorized surface may include atleast two colored reference elements with a same color and a same shadeand at least three, at least four, or at least five reference elementswith differing shades of a same color. Consistent with the presentdisclosure, the known levels of shade of the plurality of coloredreference elements may be compared with the perceived colors in acaptured image to determine the effects of local illumination conditionsand/or the effects of image capturing settings on the at least onereagent. For example, when the plurality of colored reference elementsare arranged adjacent one to another, the system may determine abrightness map of colored reference elements to calculate a correctionfactor for correcting the level of shade of a depiction of at leastcolored test reagent pad. In one example, a machine learning model maybe trained using training examples to determine correction factors forcorrecting levels of shade of depictions of reagent pad in images fromresults of comparisons of the known levels and measured levels of shadesof reference elements in the images. The trained machine learning modelmay be used to analyze the comparison of the known and measured level ofshades of reference elements in an image to determine the correctionfactor for correcting a level of shade of a depiction of a reagent padin the image. An example of such training example may include a resultof a comparison of the known levels and measured levels of shades ofreference elements in an image, together with the desired correctionfactor to be determined. Further, in some examples, the system may usethe calculated correction factor to correct the level of shade of adepiction of at least colored test reagent pad, for example bymultiplying one or more color components of one or more pixels of thedepiction of the reagent pad by the correction factor, by multiplyingone or more color components of a representative color of the depictionof the reagent pad by the correction factor, by adding the correctionfactor to one or more color components of one or more pixels of thedepiction of the reagent pad, by adding the correction factor to one ormore color components of a representative color of the depiction of thereagent pad, and so forth. In some embodiments, in response to firstdetermined local illumination conditions, the processing device mayinstruct the user to change at least one illumination aspect andrecapture an image of the of the reagent. For example, the processingdevice may indicate that local illumination conditions are too dark forcompleting the test. However, in response to second determined localillumination conditions, the processing device may forgo instructing theuser to change the at least one illumination aspect.

At step 806, the processing device may use the determined localillumination conditions and/or the determined effects of the imagecapturing settings together with an analysis of a depiction of theplurality of colored test reagent pads in the image to determine anextent of a chemical reaction on the reagent. Consistent with thepresent disclosure, determining the extent of a chemical reaction mayinclude color calibration of the plurality of colored test reagent padsin the image based on the local illumination conditions and/or thedetermined effects of the image capturing settings. In one example,color calibration may include rectifying the received image by changingcolors of pixels associated with a least one colored test element of thereagent pad based on the determined local illumination conditions and/orthe determined effects of the image capturing settings. By way ofexample, in response to a first set of determined local illuminationconditions and a first depiction of the plurality of colored testreagent pads in the image, the processing device may determine a firstextent of the chemical reaction on the reagent; and in response to asecond set of determined local illumination conditions and the firstdepiction of the plurality of colored test reagent pads in the image,the processing device may determine a second extent of the chemicalreaction on the reagent. The second extent of the chemical reaction maydiffer from the first extent of the chemical reaction. In anotherexample, in response to a first set of determined effects of the imagecapturing settings on the image, the processing device may determine afirst extent of the chemical reaction on the reagent; and in response toa second set of determined effects of the image capturing settings onthe image, the processing device may determine a second extent of thechemical reaction on the reagent. The second extent of the chemicalreaction may differ from the first extent of the chemical reaction.

In some embodiments, the processing device may generate a set ofcorrection factors based on the determined local illumination conditionsand/or the determined effects of the image capturing settings. The setof correction factors may include a specific correction factor for eachof the plurality of colored test reagent pads or a general correctionfactor for all of the plurality of colored test reagent pads. The set ofcorrection factors may include a combination of correction factorsassociated with the determined local illumination conditions andcorrection factors associated with the determined effects of the imagecapturing settings. The processing device may use the set of correctionfactors and an analysis of a depiction of the plurality of colored testreagent pads in the image to determine an extent of a chemical reactionon the reagent irrespective of local illumination conditions andirrespective of effects of the image capturing settings. In one example,the colors of the reference elements depicted in the image and the knowncolors of the reference elements may be used to determine the correctionfactors, for example using any color calibration algorithm. For example,such correction factor may include sensor response curves of the cameraused to capture the image for different colors, and the sensor responsecurves may be used to determine the actual color of a reagent pad fromthe color of the reagent pad depicted in the image.

At step 808, the processing device may cause the mobile communicationsdevice to provide to the user an indication that the testing of thereagent is complete. In one embodiment, providing the user with anindication that the testing of the reagent is complete includesproviding data indicative of the extent of the chemical reaction.Additionally or alternatively, providing the user with an indicationthat the testing of the reagent is complete includes providing a messageindicative of progress of the testing. Consistent with the presentdisclosure, the processing device may determine that the reagent isimproperly placed in the region for placement of the reagent (e.g.,dipstick placement region 460) and provide an associated notification toa user. In some cases, the notification may include an instruction torecapture an image of the reagent within a time period, e.g., within twominutes.

FIG. 9 depicts an exemplary color board 900, which may be used inreagent strip testing. Color board 900 may be, for example, a surfacethat displays various colors, areas, symbols, patterns, etc. Such asurface may be made of any combination of suitable materials fordisplaying colors, including but without limitation, paper, cardboard,plastic, cloth, wood, and/or fibers. Color board 900 may include acolorized surface 902, having a number of reference elements thereon.Some of these reference elements may be colored, such as coloredreference elements 908, which may be a part of a group of coloredreference elements, such as 470A and/or 470B, discussed with respect toFIG. 4B. In some embodiments, multiple color reference elements 908 mayshare substantially the same color and/or hue. For example, one colorreference element 908 may have a similar hue but a different chromavalue relative to another color reference element 908. As anotherexample, one color reference element 908 may have a similar hue but adifferent chroma value relative to another color reference element 908.Color elements sharing substantially the same color may be adjacent toeach other on color board 900 (e.g., on the same row within colorizedsurface 902). Other reference elements may be cube-like grey elements,which may be used for color correction, among other purposes. These greyelements may be the same as or similar to the grey elements discussedwith respect to FIG. 4B.

FIG. 9 provides but one example. The number and arrangement of elementson a color board can be configured in any number of ways. For example,while color board 900 presents an analysis region 104 on one side of allreference elements, the color board could be constructed with referenceelements on both sides of the analysis region 904, or even above and/orbelow analysis region 904. As another example, color board 900 mayinclude an analysis region 904 shaped similarly to the shape of a testobject, which may be surrounded by a colorized surface 902. Color board900 may also have multiple analysis regions or multiple unique codes906, which may enable a machine to read information associated withdifferent data.

In some embodiments, analysis region 904 may be configured to receive areagent pad. For example, analysis region 904 may be similar in size toa reagent pad and/or may include a printed outline of a reagent pad(e.g., to guide placement of the reagent pad within analysis region904). Analysis region 904 may also be made of a combination of materialssuited to receive a reagent pad. For example, analysis region 904 maycomprise materials for holding or guiding a reagent pad, such as plastic(e.g., plastic having snaps for securing a reagent pad to analysisregion 904), statically charged material, magnets, etc. Analysis region904 may also be configured in similar manners to receive a test objectother than a reagent pad, such as a dipstick, liquid, photograph (e.g.,of a wound), part of a human body (e.g., a portion of wounded skin),and/or any object presented for analysis with color board 900. In someembodiments, analysis region 904 may be transparent or may be negativespace, in order to appropriately accommodate a test object (e.g., partof a human body).

Disclosed embodiments may include a unique code on the color board, thecode reflecting specific chromatic properties associated with each ofthe first colored element and the second colored element at the time ofprinting, and wherein the code is machine readable to enable a machineto later normalize a comparison color, for determining chromaticproperties of the at least one reagent pad. The unique code may includeone or more of numbers, an alphanumeric characters, a barcode, a QRcode, a visual pattern, a pattern of punches, a pattern of embossing, anon-visual machine readable indicator such as an electronic tag, (NearField, RFID) and/or any unique visual indicator. Such a code is uniquein that it is correlated to the specific colors on the color board withwhich it is associated. If many color boards are printed with preciselythe same colors, multiple boards may share a common unique code. Theunique code may also be unique to any combination of the color board, ageographical region, a production line, a creator of the color board, ahealth care provider, a type of desired medical test and/or any of theitems discussed in the examples below.

By way of example only, color board 900 in FIG. 9, may include a uniquecode 906. In some embodiments, unique code 906 may reflect specificchromatic properties associated with any number of colored referenceelements 908 at the time of a printing of color board 900 and/orcolorized surface 902. In some embodiments, these chromatic propertiesmay include variations in colors of any number of colored referenceelements 908. For example, a chromatic property may be a variation inthe hue of a colored reference element 908. Variations in color may beexpressed relative to another colored reference element 908, or relativeto a reference that is not on color board 900. In some embodiments,unique code 906 may appear on a test object (e.g., a dipstick) ratherthan on color board 900.

Unique code 906 may be machine readable to enable a machine to normalizea comparison color. Unique code 906 may include any combination of anumber, an alphanumeric sequence, a barcode, a QR code, a unique visualpattern, a unique pattern of punches, a unique pattern of embossing,and/or any unique visual indicator. In some embodiments, unique code 906may be unique to any combination of the color board, a geographicalregion, a production line, a creator of the color board, a health careprovider, a type of desired medical test (urinalysis to determinepregnancy, urinalysis to determine a bladder infection, a spit test todetermine a mouth infection, etc.), attributes of a user of the colorboard 900, specific chromatic properties, a range of chromaticproperties, and the like. In some embodiments, unique code 906 may beconfigured to enable a machine to determine a detail regarding a testingkit with which color board 900 is associated. In some embodiments, sucha detail may be used during the execution of testing of a reagent pad(e.g., a reagent pad analyzed using color reference elements 908). Forexample, such a detail may include a geographical region, a healthmaintenance organization (HMO), a service provider, an identification ofa doctor, an identification of a patient, and/or doctor orderinformation. Based on any combination of details, including a singledetail, analysis results may be tailored to include or exclude specificinformation (e.g., doctor order information may indicate that aparticular analysis should be performed on a reagent pad).

Unique code 906 may also be configured to enable a machine to determineinformation related to a previous use of color board 900. Such previoususe information may include how many times color board 900 waspreviously used, when a previous use of color board 900 took place, atype of use, a type of analysis provided, etc. For example, a machinemay read unique code 906 and determine, based on stored data (e.g.,stored in a database), that a machine previously read unique code 906and/or that analysis related to color board 900 was provided (e.g.,based on a previously received image of color board 900). This previoususe information may be used to determine if color board 900 was usedmore than a particular number of times (e.g., three times), or if colorboard 900 was used outside a particular time window (e.g., whether colorboard 900 was used within a week of a first use).

Chromatic properties may include, but are not limited to, red-blue-greenvalues, saturation, contrast, tint, shade, hue, value, and/or chroma.Chromatic properties may also be expressed a difference betweenchromatic properties of different elements (e.g., a reagent pad andcolored reference elements 908). In some embodiments, chromaticproperties may be discernable from unique code 906 itself (i.e., uniquecode 906 indicates parameterized values of chromatic properties). Insome embodiments, chromatic properties may be determined by comparingunique code 906 to data stored in a data structure, either in additionto or as an alternative to being discernable from unique code 906itself. For example, unique code 906 may include a unique identificationnumber (e.g., an alphanumeric sequence) that corresponds to dataincluding chromatic properties (e.g., chromatic properties associatedwith a color board 900 of unique code 906).

FIG. 10 schematically depicts a color analysis system 1000, which mayinclude elements of system 100. In some embodiments, color analysissystem 1000 may include a mobile communications device 115 that maycommunicate with communications network 150, such as by sending image1002 to communications network 150. Mobile communications device 115 maybe operated by a user 110. User 110 may send image 1002 from mobilecommunications device 115 to communications network 150.

Image 1002 may include a colorized surface 132 (which may be a colorboard 900 or a portion thereof), and/or an object to be examined 134(i.e., a test object). In some embodiments, image 1002 may have beencaptured using an image capturing component of mobile communicationsdevice 115. In other embodiments, image 1002 may be captured and/or sentby a device other than mobile communications device 115, such as adesktop computer, special purpose computer, or other computing device. Adevice that captures or sends image 1002 may be associated with a user110, a medical caregiver, or entity (e.g., an urgent care center,doctor's office).

Images 1002 received at communications network 150 may be forwarded to amedical analysis unit 140, which may include a server 145 coupled to oneor more physical or virtual storage devices such as a database 146.Server 145 and/or database 146 may contain programs, rules,applications, instructions, etc. used to process image 1002 and performmedical analysis on information obtained from image 1002. For example,medical analysis unit 140 may carry out process 1100, described withrespect to FIG. 11 below. Although medical analysis unit is illustratedin FIG. 10 as a box including multiple components, all components do notneed to reside in the same location.

FIG. 11 is a flowchart of an example method 1100 for analyzing colors,executed by a processing device of system 100, such as mobilecommunications device 115, mobile communications device 125, server 145(i.e., remote from mobile communications device 115), or communicationsdevice 165, according to embodiments of the present disclosure. Forpurposes of illustration, in the following description, reference ismade to certain components of system 100. It will be appreciated,however, that other implementations are possible and that othercomponents may be utilized to implement the exemplary method (includinga portion of the exemplary method). It will also be readily appreciatedthat the illustrated method can be altered to modify the order of steps,delete steps, or further include additional steps. While the stepsdescribed below reference certain illustrated structures, it is to beunderstood that the steps are not limited to any particular structure.

Disclosed embodiments may include receiving from an image sensorassociated with a mobile communications device an image of a reagent padin proximity to a color board having a first colored reference, a secondcolored reference, a test region on the color board surface configuredto receive at least one reagent pad, and a unique code. The receipt mayoccur within the mobile communications device itself, or remote from themobile communications device, via a network data transfer. Many examplesof mobile communications devices, reagent pads, images, color boards,colored references and unique codes are described herein, and are allintended to be incorporated within this step of receiving an image of areagent pad.

By way of example, at step 1102, a processing device (e.g., a processingdevice 202 of a medical analysis unit 140) may receive an image of areagent pad. In some embodiments, the reagent pad may have been used(for example, liquid, such as urine, may have been deposited on thereagent pad so that a chemical reaction may take place). In someembodiments, the processing device may receive an image of a dipstick(e.g., a dipstick used in place of reagent pad), color board 900, and/ora test object.

At step 1104, the processing device may detect a unique code based onthe received image. For example, the processing device may detect that acolor board 900 may be in the image, and may detect a unique code 906 onpart of the color board 900. In some embodiments, the unique code itselfmay be visible in the image (e.g., an alphanumeric sequence). In someembodiments, the unique code may not visible from the image itself butmay be obtained by the processing device reading information from theimage or using a tag sensor associated with the mobile communicationsdevice. For example, the processing device may read a QR code from theimage, and the QR code may be linked to a unique code (e.g., the QR codemay point to a unique code stored in a database, which the processingdevice may query to receive the unique code).

Disclosed embodiments may also include determining from the unique codespecific chromatic properties associated with each of the first coloredelement and the second colored element. Determining the specificchromatic properties may occur in any one of many ways, including, forexample performing a look up in a table or a data structure, whichcorrelates the code with chromatic properties. Alternatively, the codeitself may encode the specific chromatic properties.

By way of example, at step 1106, a processing device may determinechromatic properties corresponding to the unique code detected at step1104. The chromatic properties corresponding to the unique code may beassociated with a colorized surface 902, which may be part of the samecolor board 900 as the detected unique code (or a visual indicator ofthe unique code). For example, the determined chromatic properties mayinclude hue, value, and/or chroma information for reference elements ofa colorized surface 902. These chromatic properties may have beenestablished at a time when the color board 900 or colorized surface 902was printed, shortly after printing, or a priori. In some embodiments,different sets of chromatic properties may be determined based on theunique code, which may be associated with different reference elements.

In some embodiments, at step 1106, information other than chromaticproperties may be determined from the unique code, which may occurbefore, after, or while determining the chromatic information. Thisother information may be related to a previous use of a color board 900(i.e., a color board 900 from which the unique code is read). Suchprevious use information may include how many times color board 900 waspreviously used, when a previous use of color board 900 took place, atype of use, a type of analysis provided, etc. For example, a processingdevice performing process 1100 may read unique code 906 and determine,based on stored data, that a processing device previously read uniquecode 906 and/or that a type of analysis related to color board 900 wasprovided (e.g., based on a previously received image of color board 900having a test object in analysis region 904). In some examples, a colorcorrection function tuned to the particular color board may be obtainedbased on the unique code. Further, in one example, this color correctionfunction tuned to the particular color board may be used to analyze theimage and correct colors in images, for example according to step 1108.In some examples, a color correction function tuned to the particularcolor board for determining the extent of a chemical reaction on areagent pad may be obtained based on the unique code. Further, in oneexample, this function may be used to analyze the image and determinethe extent of a chemical reaction on a reagent pad, for exampleaccording to step 1108.

The previous use information may be used to determine if color board 900was used more than a threshold number of times (e.g., three times)and/or if color board 900 was used outside a particular time window(whether color board 900 was used within a week of a first use, whethercolor board 900 was used within a number of months after its production,etc.). The threshold number of times or particular time window may bebased on the unique code, a type of a past usage of the color board 900,patient information, etc. In some embodiments, the threshold number oftimes or particular time window may be selected by a manufacturer of thecolor board, a medical provider, a doctor, and/or another entityassociated with the color board 900. In some embodiments, a processingdevice performing process 1100 may proceed to determine chromaticproperties, analyze a received image, or perform another part of process1110 if it determines that a number of previous uses of the color board900 does not exceed a threshold number of times.

If the processing device performing process 1100 determines that anumber of previous uses of the color board 900 exceeds a thresholdnumber of times, it may not perform another step, such as determiningchromatic properties, analyzing a received image, returning analysisresults, etc. In some embodiments, a processing device may generate anotification based on determining that the number of previous usesexceeds a threshold. For example, mobile communications device 115 maygenerate and display a notification to inform a user that the colorboard 900 may not be used again because it has already been used athreshold number of times. As another example, another device, such asserver 145, may generate a notification and send it to mobilecommunications device 115. In some embodiments, a notification may begenerated even if a use of a color board 900 does not exceed a thresholdnumber of uses, to display information to a user (a notification of atime window remaining, a number of uses remaining, medical informationrelated to a previous use, etc.). The same or similar actions may beperformed based on whether a previous use of the color board 900 waswithin or outside a particular time window, rather than, or in additionto, whether a previous number of uses exceeded a threshold.

Disclosed embodiments may include using the specific chromaticproperties of first colored reference element and the second coloredreference element to analyze a depiction of the at least one reagent padin the image for determining an extent of a chemical reaction on the atleast one reagent pad. The specific chromatic properties of thereference elements may be used, as described earlier, to generate acorrection factor or other normalization data to account for variationsthat might occur as the result of a local illumination condition or acamera setting.

By way of example, at step 1108, a processing device may analyze thereceived image. This analysis may be based on the chromatic propertiesdetermined at step 1106 and/or a part (or entirety) of the imagereceived at step 1102. For example, the processing device may comparethe determined chromatic properties to chromatic properties of a reagentpad, dipstick, and/or other test object that may be depicted in thereceived image. In some embodiments, the processing device may normalizea comparison color (a color reference element, a color of a test object,etc.) using the chromatic properties determined at step 1106, forexample as described above in relation to methods 700 and 800.Normalizing a comparison color may include removing local illuminationeffects (i.e., illumination effects distorting chromatic information ofreference elements), which may be accomplished in part by using greyreference elements, such as those discussed with respect to FIG. 4B. Insome embodiments, normalizing a comparison color may include changing achromatic property, which may be a chromatic property of a colorreference element, a test object, etc.

As part of the analysis, processing device may use specific chromaticproperties of colored reference elements to analyze a depiction of atest object (e.g., a reagent pad) in the image for determining an extentof a chemical reaction on the test object. Different sets of chromaticproperties may be used to determine different extents of a chemicalreaction on the test object. For example, a first set of chromaticproperties determined from the unique code may be analyzed relative to adepiction of the test object to determine a first extent of a chemicalreaction on the test object. Continuing with this example, a second setof chromatic properties determined from the unique code may be analyzed,in some embodiments together with a depiction of the test object, todetermine a second extent of a chemical reaction on the test object,which may differ from the first extent. Any number and combination ofchromatic properties may be determined and used to determine an extentof a chemical reaction on a test object.

In some embodiments, the type of analysis performed on the receivedimage may depend on information contained in the unique code. Forexample, a processing device may determine from reading the unique codethat particular chromatic properties, parts of a test object, and/orreference elements should be used for the analysis. As a further exampleof this aspect, a processing device may determine from reading theunique code that only some portions of a dipstick (i.e. a test object)depicted in an image are relevant to the analysis.

At step 1110, the processing device may provide analysis. In someembodiments, the analysis may include results of the analysis performedat step 1008. For example, the provided analysis may include the extentof a chemical reaction on a test object. In some embodiments, theprovided analysis may include information derived from an extent of achemical reaction on a test object. For example, based on an extent of achemical reaction on a dipstick, the processing device may determinethat an individual (i.e., an individual who has used the dipstick) has aspecific disease, nutrient deficiency, medical condition, is healthy,etc. In some embodiments, a confidence level (e.g., 90% certainty, ‘lowcertainty’, ‘high certainty’, etc.) associated with the derivedinformation may also be provided together with the provided analysis. Insome embodiments, while several analysis results may have beendetermined at step 1008, only a subset of these may be provided at step1110. For example, a type of medical information to be provided may beselected based on the unique code determined at step 1104, which maycontain information indicating that the analysis provided should belimited or expanded based on any combination of the color board 900, ageographical region, a production line, a creator of the color board900, a health care provider, a type of desired medical test (urinalysisto determine pregnancy, urinalysis to determine a bladder infection, aspit test to determine a mouth infection, etc.), attributes of a user ofthe color board 900, specific chromatic properties, a range of chromaticproperties, and the like. The processing device may then provideanalysis based on this selected type of medical information and based ona determined extent of a chemical reaction on the test object. Differenttypes of medical information may be provided, either simultaneously orat different points in time. For example, one type of medicalinformation may be provided to a user, while another type may beprovided to a medical professional. As another example, one type ofmedical information may be provided based on a user's geographic regionat one time, but at another time the user's geographic region may bedifferent, which may prompt the providing of a second type of medicalinformation. In yet another example, the unique code determined at step1104 may be used to determine a geographic region of the user, one typeof medical information may be provided to a user from a first geographicregion, and a second type of medical information may be provided to auser from a second geographic region. In an additional example, theunique code determined at step 1104 may be used to determine a serviceprovider (such as an insurer, a medical care provider, etc.) of theuser, one type of medical information may be provided to a userassociated with a first service provider, and a second type of medicalinformation may be provided to a user associated with a second serviceprovider.

The analysis provided at step 1110 may be provided to any number ofdifferent devices, including, but not limited to, mobile communicationsdevice 115, mobile communications device 125, server 145, communicationsdevice 165, communications device 175, and/or a computing device ofpharmacy 180. For example, data representing the extent of a chemicalreaction may be generated at one device (e.g., mobile communicationsdevice 115) and may be sent to another device (e.g., a medical entityassociated with a user of the mobile communications device, such as adevice associated with the user's doctor). This may be doneautomatically or based upon user interaction at the sending device(e.g., selection of graphical user interfaces that select routinginformation for the sending of the data).

As part of step 1108, 1110, or another step in process 1100, a billingaccount may be selected based on the unique code. For example, a uniquecode may contain patient identification information, which may be linkedto a billing account. In some embodiments, a billing account may beselected based on user inputs received at a device, such as mobilecommunications device 115. Information related to the billing accountmay be stored on any combination of mobile communications device 115, adatabase of insurance company 170, and/or any other device of system100. In some embodiments, the selected billing account may be updatedbased on the analysis performed on the received image, the analysisprovided, the number of times a color board 900 has been used, etc.

Disclosed embodiments may include urinalysis home testing kit. Such akit may include a plurality of elements to enable testing of urineeither at home or in any other environment.

One element in the kit may be a container configured to contain a urinesample. The container may have any form. It may have a fixed cavity sizeor a variable cavity size. For example, the container may be acollapsible cup or crucible, or any structure capable of holding afluid.

The kit may also include a dipstick including a plurality of testreagent pads thereon for measuring differing urinary properties. Thedipstick may have any shape or form and be made of any material capableof supporting test reagent pads. The reagent pads can be made of anymaterial capable of providing analytic indications, upon exposure to abiological fluid (e.g., urine). Some non-limiting examples of suchdipsticks may include dipstick 450, dipstick 500, dipstick 505, dipstick510, and so forth.

The kit may further include a blot pad for removing excess urine fromthe dipstick after being dipped in urine, to thereby enablenon-distorted image capture of the plurality of reagent pads by an imagesensor. The blot pad may include, for example, any suitable absorptivematerial capable of absorbing urine to an extent the absorptioneliminates or significantly reduces distortion in image capture thatmight otherwise occur if the dipstick is not blotted before imagecapture.

A colorized surface may also be included in the kit. The colorizedsurface may include a dipstick placement region. The dipstick placementregion may be an area on the colorized surface on which the dipstick maybe placed. The dipstick placement region itself need not be colorized,and may be characterized by markings or an outline indicating to a userthat the region is reserved for dipstick placement. Some non-limitingexamples of such colorized surfaces may include colorized surface 132,color board 900, and so forth.

The colorized surface may include a plurality of colored referenceelements greater than a number of the plurality of test reagent pads,for enabling color normalization of the plurality of test reagent padsusing the plurality of colored reference elements. These referenceelements may be disbursed across the colorized surface to enable throughimage analysis comparison, effects of local lighting conditions.

FIG. 12 depicts a urinalysis kit 1200. Though use in other locations arepossible and are within the scope of this disclosure, urinalysis kit1200 may be designed for use by an individual for a urinalysis test at ahome. Moreover, while urinalysis kit 1200 may be described in a contextof analyzing urine, other liquids (e.g., biological fluids) and/orobjects may also be analyzed using elements and processes describedherein.

In this exemplary embodiment, urinalysis kit 1200 contains a color board1202, which may be used for analyzing a dipstick (e.g., at least onetest reagent pad on a dipstick 1206). In some embodiments, color board1202 may appear the same or similar to color board 900, as described inconnection with FIG. 9. For example, color board 1202 may be a surfacethat may include reference elements on a colorized surface 1210, ananalysis region 1212, and/or a unique code 1214, which may be the sameas or similar to colorized surface 902, analysis region 904, and uniquecode 906, respectively. For example, unique code 1214 on color board1202 may be configured to enable identification of properties ofanalysis of a test object (e.g., urinalysis performed on a dipstick),consistent with the disclosed embodiments. At least one referenceelement on color board 1202 may be associated with a test reagent pad(e.g., a test reagent pad on a dipstick 1206). In some embodiments,color board 1202 may appear the same as or similar to colorized surface132. In some embodiments, at least two reference elements may beassociated with a single reagent pad. This may allow for multiple colorcomparisons for a reagent pad, which may provide for more accurateanalysis.

In some embodiments, analysis region 1212 may be configured to receive adipstick 1206. For example, analysis region 1212 may be based ondipstick 1206 (e.g., is a similar size or shape as dipstick 1206). Insome embodiments, a color board 1202 may have a plurality of colorreference elements on a first side of an analysis region 1212, and mayhave another plurality of color reference elements on a second side ofthe analysis region 1212. The analysis region 1212 may also have adistinct color, shape, texture, etc. to differentiate it (e.g., to ananalysis device) from colorized surface 1210 and/or a test object. Forexample, analysis region 1212 may be of a darker color or a lightercolor than dipstick 12016 and/or analysis region 1212. As anotherexample, analysis region 1212 may have a unique border around it.However, as with color board 900, in a broadest sense, this disclosureis not limited to a particular configuration, as a variety ofconfigurations are possible and are within the scope of this disclosure.

Urinalysis kit 1200 may also include a container 1204, which may beconfigured to contain a urine sample. Container 1204 may be made fromplastic, paper, or any material or combinations of materials capable ofretaining fluid. In some embodiments, the interior of container 1204 maybe coated with an impermeable material. In some embodiments, container1204 may be transparent or translucent, which may allow a user to see acolor of a liquid within container 1204. While container 1204 may take avariety of shapes and sizes, in some embodiments it may have a generalprofile of a tapered cup, such that it has an opening at its top that islarger than its base. In other embodiments, container 1204 may have anopening that is smaller than its base, which may reduce the risk ofspillage. Container 1204 may also include a removable lid, which may beclosed by a user to reduce the risks of spillage and/or inadvertentcontact with a contained liquid. While container 1204 may be configured(e.g., by having an appropriate size, shape, composition, etc.) tocontain any quantity of a liquid or semi-liquid, in some embodiments,container 1204 may be configured to contain between 75 mL and 750 mL ofurine. In some embodiments, container 1204 may have a fill line or otherindicator on it that indicates a proper amount of liquid for a user toput in the container. The fill line may be sized to correspond to a sizeof the dipstick, to thereby ensure that when dipped, the liquid coverseach reagent pad.

In some embodiments, container 1204 may have an adjustable size, whichmay be accomplished in a number of ways. For example, container 1204 mayhave a series of ridges (such as those shown in FIG. 12), which mayallow the walls of container 1204 to compress, such as when sufficientforce is applied. In this example, force may also be applied to stretchcontainer 1204 back to a larger size. As another example, container 1204may have a number of alternating flexible and less flexible regions thatallow a portion of container 1204 to be collapsed to decrease its size.In some embodiments, urinalysis kit 1200 may also include a funnel toassist a user in depositing a liquid into container 1204. Such a funnelmay also be of adjustable size as container 1204 and/or may be made ofsimilar materials.

Urinalysis kit 1200 may also include a dipstick 1206, which may have anynumber of test reagent pads (shown as exemplary squares on dipstick 1206in FIG. 12) for measuring differing properties (e.g., urinaryproperties). In some embodiments, dipstick 1206 may have one or moretest reagent pads that measure an albumin to creatinine ratio (i.e.,based on the degree of a chemical reaction on and/or in the pad). Forexample, dipstick 1206 may include a first test reagent pad thatmeasures a first property associated with albumin concentration, and mayinclude a second test reagent pad that measures a second propertyassociated with creatinine concentration. Other test reagent pads maymeasure other properties, such as a property associated with leucocytesin urine, a property associated with blood in urine, and/or a propertyassociated with nitrite in urine. As yet another example, a test reagentpad may be configured to measure a property associated with a urinarytract infection. Any combination of properties may be measured dependingon the number and configuration of test reagent pads on a dipstick 1206.

Different reagent pads may exhibit different colors or shades, dependingon the test liquid to which they are exposed. Test reagent pads may belabeled and/or ordered to identify their testing purposes to a userand/or a processing device. For example, a reagent pad may have a labelof “albumin concentration test.” As another example, the test reagentpads may be ordered based on a predefined order, which may be determinedand/or confirmed by a processing device after reading unique code 1214.

Urinalysis kit 1200 may also include a blot pad 1208 for absorbingliquid. In some embodiments, blot pad 1208 may be configured to absorbliquid from any number of test reagent pads on a dipstick 1206 (e.g.,after a dipstick 1206 is dipped in urine contained in container 1204).Blot pad 1208 may have an absorption capacity that is based on a size ofat least one reagent pad, an absorption capacity of at least one testreagent pad, a type of liquid being used for testing, and/or a number ofreagent pads on a dipstick 1206. By way of example, blot pad 1208 mayhave an absorption capacity of between 0.5 mL to 5 mL. In someembodiments, blot pad 1208 may be attached to a package of urinalysiskit 1200, which may allow for ease of manufacturing. In otherembodiments, blot pad 1208 may be attached to color board 1202. As ablot pad 1208 may differ based on a color board 1202 and/or urinalysiskit 1200 with which it is associated, attaching the blot pad 1208 to thecolor board 1202 or urinalysis kit 1200 may help ensure that the blotpad stays with a corresponding correct color board 1202 or urinalysiskit 1200.

FIG. 13 is a schematic illustration of a urinalysis kit image processingsystem 1300, which may be used to process an image from a device andprocess a sequence of instructions based on subsequent actions taken atthe device. In some embodiments, urinalysis kit image processing system1300 may be used in connection with a mobile communications device 115that may communicate with communications network 1350 (which may beidentical or similar to communication network 150), such as by sendingimage 1330 to communications network 1350. Mobile communications device115 may be operated by a user 110. User 110 may send image 1330 frommobile communications device 115 to communications network 1350.

Image 1330 may include a depiction of at least part of urinalysis kit1334, which may be an instance of urinalysis kit 1200. Image 1330 mayalso include a depiction of a visual link 1332, which may be attached tourinalysis kit 1334. Either though detection of unique characteristicsof the kit and/or the visual link 1332, image 1330 may be used toconfirm the identity of the test being performed. In some embodiments,visual link 1332, which may be the same as or similar to unique code906, may include any combination of a number, an alphanumeric sequence,a barcode, a QR code, a unique visual pattern, a unique pattern ofpunches, a unique pattern of embossing, and/or any other visualindicator. Visual link 1332 may be machine-readable, and when read by aprocessing device, visual link 1332 may prompt certain operations totake place at mobile communications device 115, communications network1350, and/or other processing devices. For example, visual link 1332,when read by a processing device, may prompt the initiation of process1400, as will be described later in greater detail.

Visual link 1332 may include encoded information, which may be relatedto features of unique code 906, including, without limitation, a colorboard identifier, a geographical region, a production line, a creator ofthe color board, a health care provider, a type of desired medical test(urinalysis to determine pregnancy, urinalysis to determine a bladderinfection, a spit test to determine a mouth infection, etc.), attributesof a user of urinalysis kit 1200 (or of any part of urinalysis kit 1200,of a color board such as color board 900, of a dipstick, etc.), specificchromatic properties, a range of chromatic properties. Any of thisinformation may be encoded directly in visual link 1332, or visual link1332 may direct a device to any of this information contained in anexternal location (e.g., a database or other data structure).

In some embodiments, mobile communications device 115 may send image1330, visual link 1332, and/or information associated with any of visuallink 1332, color board 1202, container 1204, dipstick 1206, blot pad1208, and urinalysis kit 1200 to communications network 1350. In someembodiments, communications network 1350 may send the informationreceived from mobile communications device 115 to instructions provider1340, which may include a server 1345 and a database 1346. In certainembodiments, instructions provider 1340 may be a medical analysis unit140 or a healthcare provider 160. Instructions provider 1340 mayinterpret the information received from mobile communications device 115and may provide information to mobile communications device 115 inresponse. For example, instructions provider 1340 may receiveinformation based on visual link 1332, which may be associated with aparticular type of desired medical test, and may provide instructionsassociated with that type of test to mobile communications device 115 inresponse. In some embodiments, these instructions may be stored atdatabase 1346. As another example, instructions provider 1340 mayreceive an image 1330 from mobile communications device 115 that has atleast one unreadable portion and in response may send a notification tomobile communications device 115 that prompts a user 110 to capture anew image 1330 and/or activates an image sensor of mobile communicationsdevice 115. Having visual link 1332 attached to a urinalysis kit 1334may help ensure that instructions for a specific urinalysis kit 1334 areprovided (for example, different urinalysis kits 1334 may be configuredfor different medical tests and therefore may have different associatedinstructions, identical urinalysis kits may be provided to differentusers that needs different instructions and/or different medicalanalysis, and so forth).

A method of this disclosure may be used for guiding a user to utilize aurinalysis home testing kit. The user may be a patient or an individualassisting a patient. The utilization may occur at home, or at any otherlocation.

Consistent with the disclosure, the user may be instructed to collect aurine sample in a container provided in the urinalysis home testing kit.This instruction (as well as all other instructions described herein)may occur through sound, text, video, graphics or a combination of oneor more of the forgoing. The manner of instruction and/or the extent ofthe instruction may vary depending on factors such as the experience ofthe user with the test. If the system recognizes the user as havingperformed the test successfully multiple time in the past, shortedinstructions may be provided.

The instruction may be provided via a server for storage on a mobilecommunications device of the user, or may be accessed in real time via aremote server in communication with the mobile communications device viaa network.

Consistent with embodiments of the invention, the user may also beinstructed to dip a dipstick provided in the urinalysis home testing kitin the urine sample. This instruction may include a time element as overor under exposure of the reagent pad may impact the accuracy of thetest. To this end, the instruction may include a mechanism for timingthe exposure of the reagent pad to the urine sample.

Also consistent with the disclosure, the user may be instructed to blotthe dipstick using a blot pad provided in the urinalysis home testingkit to remove excess urine from a reagent area, and to place thedipstick on a reference surface displaying at least two reference colorregions.

In addition, consistent with the present disclosure, the user may beinstructed to use an image sensor to capture in a common image adepiction of the blotted dipstick and the at least two reference colorregions, in order to use the at least two reference color regions as aguide to providing a test result through interpretation of the blotteddipstick relative to the at least two reference color regions. The imagecapture may occur on the same mobile communications device as used topresent the instruction to the user. The instructions may involve atimer to ensure that the image capture occurs within a prescribed timebeginning with exposure of the reagents to the urine sample. By way ofnon-limiting example, FIG. 14 depicts a process for collecting andanalyzing urinalysis information, which may be performed by anycombination of processing devices, such as mobile communications device115, server 1345, etc., consistent with the present disclosure. Any ofthe instructions described herein may comprise text, an image, ananimation, and/or a graphical user interface (GUI). In some embodiments,instructions may be displayed with selectable buttons, sliders, and/orother interactive graphical elements. At any step in process 1400 (e.g.,after an action of one of the steps has been performed), a processingdevice may cause a display of a confirmation button, and may not proceedto another step in process 1400 until a selection input is detected atthe confirmation button (e.g., a user 110 selects the button at a mobilecommunication device 115). The processing device may be remote from themobile communications device 115. For example, the processing device maybe part of a remote server that through a networked connection causesinstructions to be displayed in real time. Or the same processing devicemay cause the instructions to be transmitted to the mobilecommunications device to be stored for later display. Alternatively, theprocessing device could be part of the mobile communications deviceitself.

At step 1402, a processing device may cause a display of an instructioninstructing a user 110 to collect a urine sample. In some embodiments,this instruction may instruct the user 110 to collect a urine sample ina container 1204 that is part of a urinalysis kit 1200. In someembodiments, this instruction may include a particular amount of urineto be deposited (10 mL, ¾ of the capacity of the container 1204, etc.),a minimal required amount, a maximal required amount, and so forth. Insome embodiments, an image sensor may capture an image of container 1204and a processing device may determine, based on an analysis of theimage, whether the proper amount of liquid is in the container 1204. Forexample, a machine learning model may be trained using training examplesto determine an amount of liquids in container 1204 from images, and thetrained machine learning model may be used to analyze the image anddetermine the amount of liquids in container 1204. Further, thedetermined amount may be compared with a set of requirements (e.g., athreshold defined in memory accessible to a processing device) todetermine whether the proper amount of liquid is in the container 1204.A training example may include an image of a liquid in a container 1204together with an indicator of the amount of liquids in container 1204, aportion of an image, color information associated with an image, and/orany other data capable of training a machine to determine an amount ofliquid in a container.

At step 1404, a processing device may cause a display of an instructionto dip a dipstick 1206 in a urine sample, which may be contained incontainer 1204. Dipstick 1206 may be part of a urinalysis kit 1200. Insome embodiments, this instruction may include a desired dipping motionand/or a desired amount of time that dipstick 1206 should be held withinthe urine sample before withdrawing it. A processing device may alsodisplay a timer to a user 110 that indicates an amount of time remainingthat the user 110 should keep the dipstick 1206 in the urine sample.

At step 1406, a processing device may cause a display of an instructionto blot a dipstick 1206 on a blot pad 1208, such as to remove excessurine from a reagent area of the dipstick 1206. Blot pad 1208 may bepart of a urinalysis kit 1200. This instruction may include a degree ofpressure to be exerted between the dipstick 1206 and the blot pad 1208,an amount of time the dipstick 1206 should be held to the blot pad 1208,a number of times the dipstick 1206 should be tapped to the blot pad1208, an indication of a side of dipstick 1206 to face blot pad 1208,etc.

At step 1408, a processing device may cause a display of an instructionto place the dipstick 1206 on analysis region 1212. Analysis region 1212may be part of a color board 1202, which may be part of a urinalysis kit1200. This instruction may include information regarding a properorientation of the dipstick 1206 relative to the analysis region 1212.Analysis region 1212 may have a colorized surface 1210 next to it, whichmay depict any number of color reference elements. In some embodiments,colorized surface 1210 may have at least two reference elements. Forexample, colorized surface 1210 may have at least two reference elementsper reagent pad on a dipstick 1206, which may allow for multiple colorcomparisons of a reagent pad, which may improvement analysis. In someembodiments, analysis region 1212 may be in between two portions ofsurface 1210 such that reference elements are located on opposing sidesof the analysis region 1212.

At step 1410, an image sensor of a processing device may be activated.The image sensor may be a camera of a mobile communications device 115,an imaging device connected to a computer, or any other imager capableof capturing an image. Upon activation, the processing device eitherdirectly or through transmitted instructions, may cause an imagecaptured by the image sensor to be analyzed. The image may also bedisplayed. The processing device either directly or through previouslytransmitted instructions, may cause information to be overlaid on thedisplayed image, which may guide a user 110 to capture an image ofdesirable quality (having proper lighting, a proper orientation of anobject for analysis, etc.). For example, the processing device may causean overlay of a colored outline (e.g., a bounding box) of a color board1202, an analysis region 1212, a dipstick 1206, etc. onto a live imageseen by the image sensor. As another example, the processing device maycause a notification to a user regarding image quality to be displayed,such as an indication that the lighting is insufficient, the orientationis incorrect, etc. In some embodiments, the processing device maydetermine, such as based on a current image viewed by the image sensor,a current illumination of an object in the image (color board 1202, ananalysis region 1212, a dipstick 1206, etc.). Based on the currentillumination, processing device may directly or indirectly determinethat a currently viewed image is not desirable for analysis, and maycause the issuance of a prompt to the user 110 to take an action toadjust the illumination of the image (move the image sensor and objectto be imaged to a different location, adjust the lighting in asurrounding environment, rotate a dipstick 1206 relative to an analysisregion 1212, etc.). In some embodiments, the processing device may causeautomatic actions to adjust characteristics of the image to make it moredesirable for analysis. For example, the processing device may causeadjustment to the exposure, contrast, focus, and/or other photographicsettings to improve the quality of the image for analysis. Graphicaluser elements may be provided to a user to make these adjustmentsmanually. In some embodiments, these actions to improve image qualitymay take place at the same device capturing the image, which may, forexample, conserve processing resources of remote analysis system (e.g.,medical analysis unit 140) and increase the quality of images sent tosuch a remote analysis system. In some embodiments, the instruction mayinclude instructing a user to use a mobile communications device 115 tocapture an image of the blotted dipstick 1206 next to the colorizedsurface 1210, and may also instruct this image to be captured prior toexpiration of an image capture time window.

At step 1412, an image may be captured by an image sensor. This imagemay be captured with the image sensor that was activated at step 1410.In some embodiments, this image may be captured in response to a useraction (e.g., a user selects a graphical user interface element at amobile communications device 115). In some embodiments, the image may becaptured after the processing device has determined that a currentlyviewed image of an image sensor is of desirable quality for analysis. Inthis manner, images of poor quality may be excluded from capturing andsending to a processing device (e.g., medical analysis unit 140) foranalysis, which may save bandwidth and processing resources. Thedesirable quality may be determined by any combination of thresholdsassociated with local illumination, resolution, contrast, a number ofdetected objects for analysis (a dipstick 1206, a reagent pad, areference element, etc.), an orientation of a detected object foranalysis, an angle of the image, a distance of an object in the imagefrom an image sensor, etc. In some embodiments, the processing devicethat captured the image may send it to another device for processingand/or analysis (i.e., performing a combination of steps 1414, 1416, and1418). In some embodiments, processing device may prevent a user 110from capturing an image before the beginning of a time window. The timewindow may be based on determinations made at a processing device. Forexample, based on instructions from a remote processor, mobilecommunications device 115 may detect a selection of a confirmationbutton indicating that a user 110 has finished blotting a dipstick 1206,and the time window may begin after a particular amount of time haspassed since the finishing of the blotting. In another example, mobilecommunications device 115 may detect a selection of a confirmationbutton indicating that a user 110 has finished dipping dipstick 1206,and the time window may begin after the particular amount of time haspassed since a finishing of the dipping. In an additional example,mobile communications device 115 may instruct user 110 to being dippingdipstick 1206, and the time window may begin after a particular amountof time has passed since the instruction to dip dipstick 1206 wasprovided. In another example, mobile communications device 115 mayanalyze a video feed depicting handling of dipstick 1206 to identify anoccurrence of a particular event in the handling of dipstick 1206 (suchas a beginning of dipping of dipstick 1206, a finishing dipping ofdipstick 1206, a finishing of blotting dipstick 1206 on a blot pad,etc.), and the time window may begin after a particular amount of timehas passed since the identified occurrence of a particular event. As yetanother example, mobile communications device 115 may begin the timewindow after it has determined that a currently viewed image of an imagesensor is of desirable quality for analysis. In some embodiments, ifinstructions provided by a processing device determines that a capturedimage is not suitable for analysis, but the time window has not yetexpired, a user may be instructed to capture another image. Instructionsprovided by a processing device may also prevent a user 110 fromcapturing an image after the time window has passed. In someembodiments, a processing device may cause a notice to be provided to auser that an image capture time window is about to end (e.g., bydisplaying the notice at a display of mobile communications device 115).

At step 1414, an image of a dipstick (e.g., 1206) may be capturedtogether with a color region. The color region may include referenceelements on a colorized surface, such as colorized surface 1210. In someembodiments, the processing device may cause the detection of at leastone reagent pad and at least one color reference element, which may beassociated with an analysis test for the at least one reagent pad. Insome examples, a detected element (dipstick 1206, color referenceelement, reagent pad, row of color reference elements, etc.) may betagged with an identifier for use in analysis. For example, a colorreference element may be tagged with an identifier indicating it isassociated with a test for a creatinine concentration of a reagent pad(which may have its own identifier). In some examples, elements, such ascolor reference elements and reagent pads, may be identified based ontheir relative position within other objects, such as the relativeposition of a color reference element within a color board or therelative position of a reagent pad on a dipstick, for example asdescribed above. In some embodiments, elements such as color referenceelements and reagent pads may be identified based on their relativeposition with respect to other elements, such as position markers 474and 475. In some embodiments, the detection may be based on the distinctcolor, shape, texture, etc. of analysis region 1212, discussed withrespect to FIG. 12. In some examples, an identity of an element, such ascolor reference elements and reagent pads, may be verified based on theappearance of the element. For example, an element, such as a colorreference element or a reagent pad, may be associated with a range ofvalid colors, valid shapes, and/or valid textures, and the appearance ofthe element in the image may be analyzed to determine whether the color,shape, and/or texture of the element is within the range of validcolors, valid shapes, and/or valid textures. In one example, in responseto a determination that the color, shape, and/or texture of the elementis not within the range of valid colors, valid shapes, and/or validtextures, one or more additional steps or actions involving the kit makeoccur, such as the color board and/or the dipstick may be forgone, andin some cases, an error notification may be provided. For example, thesystem may forgo providing medical information based on the kit, thecolor board and/or the dipstick in response to the determination thatthe color, shape, and/or texture of the element is not within the rangeof valid colors, valid shapes, and/or valid textures, and may providethe medical information in response to a the determination that thecolor and/or texture of the element is within the range of valid colorsand/or valid textures.

In some embodiments, step 1414 may further include analyzing the imageto determine whether a dipstick is a valid dipstick. For example, adipstick may be an unauthorized dipstick, may include a different numberof reagent pads than required, may include different types of reagentpads than required, may include the reagent pads in a different orderthan required, may be unsuitable for a particular medical analysis, andso forth. In one example, a machine learning model may be trained usingtraining examples to determine whether dipsticks are valid from imagesof dipsticks, and the trained machine learning model may be used toanalyze the image and determine whether the dipstick is a valid dipstickfor the process. Such training examples may include an image of adipstick together with an indicator of whether the dipstick is a validdipstick, a portion of an image, color information associated with animage, and/or any other data capable of training a machine to determinewhether a dipstick in an image is valid, consistent with the disclosedembodiments. In some examples, in response to a determination that thedipstick is invalid, one or more additional steps or actions involvingthe kit may occur, such as the color board and/or the dipstick may beforgone, and in some cases, an error notification may be provided. Forexample, the system may forgo providing medical information based on thekit, the color board and/or the dipstick in response to thedetermination that the dipstick is invalid, and may provide the medicalinformation in response to the determination that the dipstick is valid.

In some embodiments, step 1414 may further include analyzing the imageto determine whether a color board is a valid color board for theprocess. For example, a color board may be an unauthorized color board,may include a different number of color reference elements thanrequired, may include different types of color reference elements thanrequired, may include the color reference elements in a differentarrangement than required, may be unsuitable for a particular medicalanalysis, and so forth. In one example, a machine learning model may betrained using training examples to determine whether color boards arevalid from images of color boards, and the trained machine learningmodel may be used to analyze the image and determine whether the colorboard is a valid color board for the process. Such training examples mayinclude an image of a color board, together with a label indicatingwhether the color board is a valid color board, a portion of an image,color information associated with an image, and/or any other datacapable of training a machine to determine whether a color board in animage is valid, consistent with the disclosed embodiments. In someexamples, in response to a determination that the color board isinvalid, one or more additional steps or actions involving the kit mayoccur, such as the color board and/or the dipstick may be forgone, andin some cases, an error notification may be provided. For example, thesystem may forgo providing medical information based on the kit, thecolor board and/or the dipstick in response to the determination thatthe color board is invalid, and may provide the medical information inresponse to the determination that the color board is valid.

At step 1416, a color on dipstick 1206 may be compared to a colorregion, such as one or more elements on a colorized surface 1210. Forexample, a processing device may cause a comparison of a color of areagent pad on dipstick 1206 to a color of one or more referenceelements in the image captured at step 1412. As another example, a colorof a reagent pad on dipstick 1206 may be compared to at least onechromatic property (e.g., a chromatic property determined according toprocess 1100). In some embodiments, a color comparison result may begenerated after at least one color comparison is made.

At step 1418, an analysis may be provided, based on the color comparisonperformed at step 1416. In some embodiments, the analysis may beperformed at one device, such as a server 145, and provided to anotherdevice, such as a mobile communications device 115. The analysis mayinclude information related to the comparison performed at step 1416.For example, the analysis may include a similarity percentage between acolor on dipstick 1206 and a color region, which may be based on thecomparison. The analysis may also include information detailing a numberof tests performed using dipstick 1206 (e.g., a test for determining analbumin concentration) and/or results associated with one or more ofthose tests. In some embodiments, the analysis may include analysisderived from the color comparison performed at step 1416. For example, aprocessing device may determine, based on any combination of any numberof color comparisons, that at least one color comparison indicates aresult for a particular condition (pregnancy, a bladder infection, amouth infection, a nutrient deficiency, etc.). Such a result may have anassociated degree of likelihood (e.g., 70% chance of pregnancy), whichmay be based on a degree of similarity between a chromatic property of acolor of a reagent pad and a reference element. In some embodiments,analysis results may be inconclusive. For example, dipstick 1206 may nothave been blotted sufficiently to produce reliable analysis results,dipstick 1206 may not have been properly dipped, etc. In someembodiments, when analysis results are inconclusive, a processing devicemay display an instruction. For example, a processing device may displayan instruction instructing a user to reblot the dipstick 1206 or blot anew dipstick 1206. As another example, a processing device may displayan instruction instructing a user to recapture an image of dipstick1206.

According to one aspect of the disclosed embodiments, systems, methods,and computer readable media are set forth that allow for the automaticupdating of a patient electronic medical record (EMR) with results froman image analysis. Such systems, methods, and computer readable mediamay utilize at least one processor. As used herein, the term “at leastone processor” refers to any structure or combinations of structurescapable of performing logic operations on data, regardless of whethersuch structure(s) are collocated or disbursed. For example, the at leastone processor may include a processor of processing device 202, aprocessor within a mobile communications device (e.g., mobilecommunications devices 115 and 125), a processor within a server (e.g.,server 145) remote from the mobile communications device, and so forth.In another example, such processors may include one or more integratedcircuits, microchips, microcontrollers, microprocessors, all or part ofa central processing unit (CPU), graphics processing unit (GPU), digitalsignal processor (DSP), field programmable gate array (FPGA), or othercircuits suitable for executing instructions or performing logicoperations. The processing device may include at least one processorconfigured to perform functions of the disclosed methods such as amicroprocessor manufactured by Intel™. The processing device may includea single core or multiple core processors executing parallel processessimultaneously. In one example, the processing device may be a singlecore processor configured with virtual processing technologies. Theprocessing device may implement virtual machine technologies or othertechnologies to provide the ability to execute, control, run,manipulate, store, etc., multiple software processes, applications,programs, etc. In another example, the processing device may include amultiple-core processor arrangement (e.g., dual, quad core, etc.)configured to provide parallel processing functionalities to allow adevice associated with the processing device to execute multipleprocesses simultaneously.

The at least one processor may include any structure capable ofperforming logic operations on data. The at least one processor canembodied in any form, such as a personal computer, laptop computer,desktop computer, tablet computer, notebooks, mobile phone, a terminal,a kiosk, PDA, a cloud-based computing device, local or remote server(s)smart phone, smart device or any other system allowing for processing ofinformation. By way of one example of an embodiment of this disclosure,in FIG. 18, processor 1801 is disclosed, exemplary operations of whichare described later in greater detail.

According to one aspect of this disclosure, the at least one processormay be configured to generate a token for a specific patient in need ofa medical test. The token may be any means to secure information beingtransmitted from one location to another. For example, the token may bea key, an identifier, a code, or any other data that aids inauthentication of a transaction or transmission. The transmission maytake place over a personal area network, local area network, wide areanetwork, global network, the Internet, communications network 150, orany wired or wireless communication path. The token may be randomlygenerated or generated based on an algorithm or series of algorithms. Adatabase, or token vault, may store the relationship between a sensitivevalue and the token.

In one example, the token may be generated for a person in need of amedical test. A specific patient may be any person in need of a medicaltest. A medical test may be a urinalysis, a saliva analysis, a stoolanalysis, a blood analysis, sweat analysis, or any test that involvesthe measurement of a physical, chemical or biological component from apatient. In another example, the token may be generated for a particulartest ordered by a particular medical care giver to a particular patient.

In one aspect, the at least one processor may be configured to transmitthe token to a mobile communications device of a specific patient, suchas mobile communications device 115. The mobile communications devicemay be capable of allowing communication between a patient and thesystem. A mobile communication device may take the form of a phone, asmart phone, a smart watch, a tablet, a laptop, a personal computer, aPDA, smart glasses, or any device which may allow for communicationbetween a patient and the system. The transmitting may take place overany wired or wireless connection, for example over communicationsnetwork 150.

By way of one example with reference to FIG. 18, the token may betransmitted to a mobile communications device 1803. For example, mobilecommunications device 1803 may be identical or similar to mobilecommunications devices 115 and 125 described above. The mobilecommunications device 1803 is illustrated with an app 1804 for aiding apatient. An example of communication (such as a token-basedcommunication) between the mobile communication device 1803 and a remoteserver 1806, is shown at step 1805. In one example, in step 1805 themobile communication device 1803 may provide and/or transmit the token,or information based on the token, to remote server 1806. Further, insome examples, in step 1805 the mobile communication device 1803 mayprovide and/or transmit image-related information (discussed later ingreater detail) to remote server 1806. For example, in step 1805, theimage-related information may be provided in a way that associates itwith the token, or with the information based on the token. Incommunication 1807 between remote server 1806 and processor 1801,verification may occur, confirming that transmitted image-relatedinformation (discussed later in greater detail) is associated with thespecific patient. In one example, in communication 1807, remote server1806 may provide and/or transmit the token, or information based on thetoken, to processor 1801. Further, in some examples, in communication1807, remote server 1806 may provide and/or transmit image-relatedinformation (discussed later in greater detail) to processor 1801. Forexample, in communication 1807, the image-related information may beprovided in a way that associates it with the token, or with theinformation based on the token. In communication 1808 from processor1801 and database 1810, an updating of an electronic medical record mayoccur in database 1810, for example based on the token and/or theimage-related information. In some examples, communication 1808 mayoccur directly from remote server 1806 to database 1810.

In one aspect, the at least one processor may enable a communicationsession (such as a token-based communications session) to take placebetween a mobile communications device of a specific patient (such asmobile communication device 1803) and at least one remove server (suchas remote server 1806). The remote server may include a single server orone of a plurality of servers. The remote server may take the form ofone or more cloud-based devices. The at least one remote server may beassociated with at least one of a healthcare provider, an agent of ahealthcare provider, a private party, a clearinghouse, a governmentagency, or any party having an interest in the managing ofhealth-related data or electronic medical records. In some examples,processor 1801 may be part of remote server 1806, while in otherexamples processor 1801 may be separate from remote server 1806. In oneexample, remote server 1806 may be similar or identical to server 145.

Disclosed embodiments may involve a communications session that includestransmission from the mobile communications device of image-relatedinformation obtained via an image sensor associated with the mobilecommunications device. A communications session may include any one-wayor multi-way transmission or group of transmissions that involves theconveyance of information. In exemplary disclosed embodiments, thetransmission may relate to one or more images. For example, theinformation may be derived from an image sensor that may include anydevice capable of detecting and converting optical signals in thenear-infrared, infrared, visible and/or ultraviolet spectrums intoelectrical signals. Such sensors may be embodied in, for example, in themobile communications device itself, or may be embodied in a deviceassociated with a mobile communications device. By way of example, animage sensor 1811 (e.g., embedded camera, image sensor 226, etc.) inmobile communications device 1803, may be used to transmit image-relatedinformation in the transmission. The image-related information may bethe image itself, a subset of the image, or information derived from theimage or a portion thereof. In some examples, in communications sessionmay associate the transmitted image-related information with the token,or with information based on the token.

In some exemplary disclosed embodiments, the image-related informationmay reflect a resulting color of a chemical reaction between abiological substance and a reagent. The biological substance may be anybiological material, such as, but not limited to, urine, mucus, sweat,breast milk, stool, interstitial fluid, secretions, or any otherbiological material that is capable of causing a reagent to changecolor. The reagent may vary depending on the desired reaction, as eachdiffering biological substance may have differing reactions withdiffering reagents. While reactions with reagents may be part ofimage-related information in some aspects of this disclosure, in otherembodiments, as is discussed in other portions of this disclosure, theimage-related information may be associated with an image that does notinvolve a reaction with a reagent.

In some disclosed embodiments, the at least one processor may beconfigured to verify or determine, based on the token or informationbased on the token, that the image-related information is associatedwith the specific patient. The token may be, for example, a series ofbits that can be verified by the receiver as having originated with anauthorized transmitter. The token, may take the form of a kind ofsecurity key, password, passphrase, automatically generated code,digital signature, biometric data, or any other form of data capable ofenabling a receiver to verify that it is receiving information from averified sender. For example, verification may be performed by comparinga token submitted or transmitted by a patient in conjunction withtransmission with a token generated by the system. In some examples, thetoken or information based on the token may identify a particularpatient and/or a particular medical care giver and/or a particularmedical test and/or a particular medical test kit (such as particularurinalysis test 1200). For example, the token or information based onthe token may include an index to an entry (e.g., in a data structureand/or a database) that comprises the identifying information, mayencode the identifying information, and so forth. Further, in someexamples, the at least one processor may be configured to determine,using the token or information based on the token, the identity of theparticular patient and/or of the particular medical care giver and/or ofthe particular medical test and/or of the particular medical test kit(such as particular urinalysis test 1200). In some examples, the atleast one processor may be configured to verify that the identifiedparticular patient and/or particular medical care giver and/orparticular medical test and/or particular medical test kit is compatiblewith the image-related information. For example, the at least oneprocessor may verify that the image-related information is based on atype of medical test kit that matches the type of the identifiedparticular medical test kit. In another example, the at least oneprocessor may verify that the image-related information includesinformation that matches the type of the identified particular medicaltest. In some examples, in response to a failure to verify that theidentified particular patient and/or particular medical care giverand/or particular medical test and/or particular medical test kit iscompatible with the image-related information and/or to a determinationthat the identified particular patient and/or particular medical caregiver and/or particular medical test and/or particular medical test kitis incompatible with the image-related information, the at least oneprocessor may forgo one or more actions (such as one additional steps)and/or forgo a particular update to an EMR.

FIGS. 19A and 19B illustrate various configurations of a systemconsistent with some embodiments of the disclosure. FIG. 19A illustratesa configuration where the server 1900 includes a token generating module1901, a token verification module 1902, a secure communication module1903, an updating module 1904, and an electronic medical record database1905. In this configuration, server 1900 communicates with a mobilecommunications device 1910, for example as described above. Each modulemay comprise software, hardware or any combination of software andhardware in order to achieve the desired function of the module. Forexample, mobile communications device 1910 may be identical or similarto mobile communications device 1803, to mobile communications devices115 and 125, and so forth. For example, server 1900 may include one ormore of processor 1801, remote server 1806, server 145, and so forth.Token generating module 1901 may generate a token which may be used bythe secure communication module 1903 in a communication session (such asa token-based communication session) 1906 between the mobilecommunications device 1910 and the server 1900. In communication session1906, server 1900 may transmit the token to mobile communications device1910, and a user may transmit image-related information to the server1900, for example together with the token. In another example, the tokenmay be provided to mobile communications device 1910 in a different way,for example the token may be provided to mobile communications device1910 through a visual code included in a medical test equipment (such asvisual link 1332 in urinalysis test 1200, unique code 1214 on colorboard 1202, unique code 906 on color board 900, unique code 515, and soforth), the at least one processer may be configured to determine thetoken from an analysis of an image of the visual code, and the token maybe transmitted from mobile communications device 1910 in associationwith image-related information to the server 1900.

In a first example, the image-related information may include an imageof a dipstick 450 adjacent or near a colorized surface 132. In such anexample, after placing the dipstick 450 in a biological fluid, the userplaces the dipstick 450 on colorized surface 132 as shown in FIG. 4B,and utilizes mobile communications device 1910 to capture an image ofcolorized surface 132 and dipstick 450. The image may then betransmitted to server 1900, where the token verification module 1902verifies that the image is associated with the user. The server 1900 mayperform an analysis of the image and calculate a test result. Updatingmodule 1904 may then update a personal electronic record found in EMRdatabase 1905 for the user showing the test result. It is noted thatmobile communications device 1910 may also (or alternatively) perform animage analysis and transmit a result of the analysis to server 1900. Insuch an alternative situation, a token may be used to verify that thetransmitted analysis originates with the appropriate patient.

In a second example, the image-related information may reflectcharacteristics of different segments of at least one skin feature ortissue feature. A skin feature or tissue feature may include a growth, amark, a scar, an injury or any blemish or element on a skin surfacewhich may be indicative of a medical condition. Other types ofimage-related information for tissue other than skin may relate to thetongue, throat, genitalia, finger or toe nails, or other non-skin tissue(collectively referred to herein generically as “skin features”). By wayof example, a user may place a colorized surface near to or adjacent askin feature and capture an image of the skin feature and the colorizedsurface. The image may then be transmitted to server 1900, where thetoken verification module 1902 verifies that the image is associatedwith the user. The server 1900 may perform an analysis of the image andupdating module 1904 may then update a personal electronic record foundin EMR database 1905 with the result of the analysis. As in otherembodiments, the mobile communications device 1910 may alternativelyperform an image analysis and transmit a result of the analysis toserver 1900.

FIG. 19B shows an alternate arrangement. As shown in FIG. 19B, server1920 may be separate from a provider server 1940 containing EMR database1945. The server 1920 may include a token generating module 1921, atoken verification module 1922, a secure communication module 1923, andan updating module 1924. In this configuration, server 1920 maycommunicate via channel 1926 with a mobile communications device 1930and may communicate via channel 1937 with provider server 1940.Additionally, provider server 1940 may directly communicate via channel1936 with mobile communications device 1930. Each module may comprisesoftware, hardware or any combination of software and hardware in orderto achieve the desired function of the module. Token generating module1921 may generate a token which may be used by the secure communicationmodule 1923 in a communication session (such as a token-basedcommunication session) via channel 1926 between the mobilecommunications device 1930, server 1920 and provider server 1940. Incommunication session via channel 1926, a user may transmitimage-related information to the server 1920. As used herein, the term“channel” refers to any pathway or combination of pathways that enablecommunication of information. For example, a channel may include asingle leg, or a combination of discrete wireless legs of one or morewireless networks, wired networks (with or without interposed servers orother equipment). Communication via channel 1926 may be similar oridentical to communication session 1906 described above.

As alternatively illustrated in FIG. 19B, updating module 1924 mayupdate a personal electronic record found in EMR database 1945 for theuser showing the test result. EMR database 1945 may be updated viamobile communications device 1930 as via communication channel 1936 orby updating module 1924 of server 1920 via communication channel 1937.

It is understood that token generating module 1921, token verificationmodule 1922, secure connection module 1923 and updating module 1924 maybe incorporated into server 1920, mobile communications device 1930,provider server 1940, or a combination thereof as desired. For instance,token generating module 1921, token verification module 1922, secureconnection module 1923 and updating module may be incorporated intomobile communications device 1930. Such a configuration may allow fordirectly updating of a user's electronic medical record from a securemobile communications device 1930. Of course, so long as verification isenabled in some way, verification may occur with structures other thanthose in the provided examples.

In one aspect, the at least one processor may be configured to update apersonal electronic medical record of the specific patient with a testresult reflective of the verified image-related information. Updating apersonal record may include the addition of any information related tothe verified image-related information. For example, a personalelectronic record may be updated to include a test result such as ananalyte concentration, images received from the mobile communicationsdevice, the result of a test (e.g., positive or negative), a result of aurinalysis, a result of a medical test performed using process 700, aresult of a medical test performed using process 800, a result of amedical test performed using process 1100, a result of urinalysisperformed using process 1400, a result of process 1700, or any otherinformation capable of being obtained from an image, metadata thereof,or other image-related information. The electronic medical record may bestored in a local database or memory, remote database or memory, or aplurality of local or remote databases or memory.

A method of securely updating personal electronic medical records 2000in accordance with one aspect of the invention is illustrated in FIG.20. The method may be capable of being carried out by at least oneprocessor or plurality of processors configured to carry out the desiredsteps. A non-transitory computer readable medium may containinstructions that when executed by the at least one processor cause theat least one processor to perform a method, such as method 2000illustrated in FIG. 20. The processor or processors may be local,remote, part of the cloud, or any combination thereof. A token may begenerated in step 2001 for a specific patient in need of a medical test.The token may be indicative of an authorization for an image-basedmedical test for the specific patient, and the at least one processorthat generates the token may be part of at least one remote server. Theat least one remote server may be associated with at least one of ahealthcare provider and/or an agent of the healthcare provider. The atleast one remote server may be located within a healthcare institutionsuch as a hospital or doctor's office. The at least one remote servermay be associated with a clearinghouse that manages health relatedinformation.

In step 2002, the token may be transmitted to a mobile communicationsdevice of the specific patient. This may ensure that only the specificpatient is able to update information related to the medical test,however in other embodiments, other authorized entities may additionallybe provided with access. A communications session (such as a token-basedcommunications session) may be enabled in step 2003. The communicationssession (such as the token-based communication session) may includesending a link from a healthcare provider to the mobile communicationsdevice associated with the specific patient after obtaining anindication that the specific patient needs the medical test. The linkmay be sent over any wired or wireless communication channel and maytake the form of an email, SMS or text message, or any other formallowing for delivery of a link to a user. Enabling a token-basedcommunication session may include sending an identifying detail of apatient to a remote server after obtaining an indication that thespecific patient needs the medical test. An identifying detail mayinclude a username, a phone number, an email, a physical address, abirthdate, photo, biometric parameter, or any detail associated with apatient which can be used to identify a specific patient. Alternatively,a test kit may include a unique code that serves as a token for enablinga verified communication session.

A communications session (such as a token-based communications session)may include causing a mobile communications device of the specificpatient to provide guidance to the specific patient on how tosuccessfully complete the medical test. Such guidance may includepresenting directions on a display of a mobile communications device forone or a plurality of different medical tests. The directions may begiven in the form of text, audio, animation, still images, or anycombination thereof. Additionally, the directions may be given in theform of audible prompts. The directions may include interactive featurescapable of responding to prompts or questions presented by the patient.The guidance may include instructions to carry out a desired medicaltest. Such tests may include the sampling or testing of a biologicalfluid such as urine, saliva, blood, interstitial fluid or stool. Theguidance may include instructions to do one or at least two of: openinga medical kit, expanding a collapsed measuring cup, dipping a dipstick,blotting the dipstick, placing the dipstick on a colorized test surface,capturing an image of a dipstick on a colorized test surface,recapturing the image of the dipstick on the colorized test surface,guidance on how to prick a finger for blood sampling, instructions toexpose a reagent to a biological fluid and to capture an image of theexposed reagent within a predefined time window, or any informationwhich may be beneficial in directing a patient to perform a testingprocedure.

Once an image is obtained with a mobile communications device of a skinsurface, a colorized surface, a reagent pad, or a combination thereof,the image-related information may be transmitted from the mobilecommunications device at step 2004. The image-related information may betransmitted to a local processing device, a memory, a server, a remoteserver or one or more cloud devices. At step 2005, the image-relatedinformation may be verified, for example based on the generated token,to be image-related information associated with the specific patient.(The order of image receipt and verification is not necessarilycritical. For example, the communications session may be verified inadvance of image receipt, in conjunction with image receipt, or afterimage receipt.) Once verification has taken place, an update to apersonal electronic medical record of the specific patient takes placeat step 2006. Updating the electronic medical record may include thetransmission and/or storage of information relating to the test. Suchinformation may include one or more test results, the transmitted image,data derived from the transmitted image, an analysis, or any combinationthereof.

In some embodiments, a token-based communications session may includeany communications session that enables the association of image-relatedinformation (for example, image-related information originating from amobile communication device) with a particular patient (for example in aserver that receives the image-related information) using a tokenconfigured to enable such association. In some examples, such token maybe generated by the server, or by a device other than the mobilecommunication device. Further, in some examples, such association ofimage-related information and particular patient may be used, forexample by the server, to update an EMR of the particular patient basedon the image-related information. In one example, the token-basedcommunications session may include a transmission of the token with theimage-related information, for example from the mobile communicationdevice to the server, while in another example, the token-basedcommunications session may include a transmission of the image-relatedinformation in a way that enables an association of the image-relatedinformation with the token, without necessarily transmitting the token,for example by transmitting the image-related information in acommunication channel associated with the token, by transmittinginformation indicative of the token with the image-related information,and so forth. In one example, the mobile communication device mayreceive the token, while in other examples the mobile communicationdevice may never receive the token.

Consistent with some embodiments of the disclosure, a home testing kitmay be distributed to a plurality of individuals with preexistingconditions, or to individuals that may be at risk for developing acertain medical condition, as part of a preventative health careprogram, or as directed by a healthcare professional. In one example,weight, diet, lifestyle, hereditary traits, geographic location, age,sex, and socioeconomic status may be factors that indicate an individualis more likely to develop a certain medical condition. The at least oneprocessor may be configured to determine that a specific patient needs(or might benefit from) a medical test, and in response to thatdetermination, trigger the sending of a home testing kit to the specificpatient. The processor may analyze or query a database of one or morerisk factors, patient characteristics, or other medical orpatient-related information to make a determination that a patientshould be sent one or more home testing kits. Additionally, the at leastone processor may query a database of patients and initiate the sendingof different types of medical tests to one, or a plurality ofindividuals randomly, at predetermined periods of time, or when promptedby an administrator or health care professional.

Once a medical test is delivered to a specific patient, the at least oneprocessor may be configured to send reminders to the specific patientwhen the transmission of image-related information is not receivedwithin a selected time period after the home testing kit was sent. Byreminding the patient, an increase in patient compliance may beachieved. The reminder may be sent to the mobile communications deviceassociated with the patient, and may take the form of a visual remindersuch as a text, email, animation or image; an audible reminder such as abeep, tone, or voicemail; a tactile reminder such as a vibration, or anycombination thereof. The selected time period may be a predeterminedtime period for all patients, but may also be individually configured toa specific patient. Criteria such as a patient's age, past behavior of apatient (i.e. whether or not a patient has previously complied or notcomplied with a home health test), health provider, geographical region,type of medical test, and so forth may be used to develop a remindertype specific to an individual. A neural network or self-learning systemmay be used to develop reminders that achieve the greatest patientcompliance for performing a home test and submitting image-relatedinformation.

The system may utilize a result from a medical test to create a promptfor a medical professional. The system may send a notification to amedical professional to bring attention to the medical professionalabout a result of a medical test. Such a notification may be useful todirect the attention of a medical professional to a result that deviatessubstantially from a norm. The at least one processor may be configuredto cause a generation of a medical prescription for treating thespecific patient based on the test result reflective of the verifiedimage-related information. The prescription may then be sent to amedical professional for authorization. Once a specific patient has beenassociated with a test result, generating a medical prescription may beused as a notification to the medical professional that the test resulthas been received. A medical prescription may include a prescription forone or more drugs, vitamins, supplements, therapy, or any recommendationfrom a health professional based on the test result.

In one aspect, the medical test may be used to determine a parameterfrom a chemical reaction between a biological fluid and a test reagent.The color of the test reagent following the chemical reaction indicatesa concentration or level of a measured parameter. The biological fluidmay be blood, saliva, urine, mucus, sweat, breast milk, stool,interstitial fluid, secretion, or any other biological fluid. Adipstick, included as part of the home testing kit, may include morethan one reagent. A plurality of test reagents may be positioned alongthe length of the dipstick for measuring differing biological fluidproperties, analytes, or parameters. During the testing procedure, oncethe chemical reactions between the reagents and the parameters havecompleted, the dipstick is positioned proximate or adjacent to acolorized test surface. An image of the dipstick with correspondingreagents and the colorized test surface may be obtained. The transmittedimage-related information may include an image of at least one testreagent adjacent a colorized test surface. The system may be configuredto analyze an image prior to transmission. The transmitted image-relatedinformation may include data derived from image analysis of at least onetest reagent adjacent a colorized test surface. Image analysis mayinclude any type of analysis of an image, a portion of the image, orimage-related information. If the medical test includes urinalysis, thetransmitted image-related information may reflect resulting colors ofmultiple chemical reactions between a biological fluid and a pluralityof test reagents for measuring differing urinary properties. Theproperties may include physical or chemical properties of the urine.Also, concentrations or amounts of various analytes, markers, compoundsor components found in the urine may be detected. If the medical testincludes saliva testing or salivaomics, the transmitted image-relatedinformation reflects resulting colors of multiple chemical reactionsbetween a biological fluid and a plurality of test reagents formeasuring differing saliva properties that may be used to screen for ordiagnose numerous conditions and disease states, including Cushing'sdisease, anovulation, HIV, cancer, parasites, hypogonadism, allergies,circadian rhythms shifts, and any other condition capable of detectionthrough saliva. The properties may include physical or chemicalproperties of the saliva. Also, concentrations or amounts of variousanalytes, markers, compounds or components found in the saliva may bedetected. If the medical test includes stool analysis, the transmittedimage-related information reflects resulting colors of multiple chemicalreactions between a biological secretion and a plurality of testreagents for measuring differing stool properties that may provideindicators of bleeding, lactose intolerance, the presence of aninfection, Steatorrhea, pancreatitis, or any other condition detectablevia stool analysis. The properties may include physical or chemicalproperties of the stool. Also, concentration or amounts of variousanalytes, markers, compounds or components found in the stool may bedetected.

In another aspect of the invention, the image-related information mayreflect characteristics of different segments of at least one skinfeature. The at least one skin feature may include one or a plurality ofa skin mark, such as a mole, nevi, tumor, scar, freckle, blemish, cut,scrape, or any imageable feature on a skin surface, and the transmittedimage-related information may be indicative of a change in visualcharacteristic of the skin mark. A visual characteristic may include asize, shape, color or any characteristic that is observable or capableof being seen or captured with an image sensor. Any feature that may becaptured on an image may be a visual characteristic. A user may capturean image of skin feature with a mobile communications device andtransmit the image to a remote server for analysis. The image mayinclude image-related information. The transmitted image-relatedinformation may be data derived from image analysis of an image of theat least one skin feature being positioned adjacent a colorized testsurface. The data may include any information related to an analysis ofthe image or a portion thereof. The skin feature may be positionedproximate or adjacent a colorized test surface when the image iscaptured such that the transmitted image-related information is an imageof the at least one skin feature adjacent to a colorized test surface.Alternatively, an image of the skin feature may be compared to aseparate image of a colorized test surface. The system utilizes thedifferent colored portions of the colorized test surface to aid in ananalysis of the skin feature. For example, the system may normalizecolors of the skin feature in the image base on the colors of thecolorized test surface in the image. Additionally, the system maymeasure skin features based on the measurements of the colorized testsurface in the image. The colorized test surface may be used as areference for a plurality of different analyses performed by the system.The system may perform an analysis based on a single image or a seriesof images over time. For example, the system may conclude a wound isinfected by analyzing a single image and the associated image-relatedinformation. Alternatively, the system may analyze a series of imagesand the corresponding image-related information of a skin feature, anddetermine based on a change in the skin feature over time that the skinfeature may be pre-cancerous or cancerous, that a wound is not healingproperly, or that some other potential abnormality exists. The at leastone skin feature may include at least a part of a wound and thetransmitted image-related information may be indicative of healingprogress of the wound. As mentioned above, this analysis may beperformed over a period of time or a series of images. The colorizedtest surface recorded concurrently with each image in a series may beused by the system to account for changes in color of the wound duringthe healing process. Additionally, such a determination may be madeagainst a benchmark image.

In another aspect, a non-transitory computer readable medium isdisclosed for enabling automatic update of an electronic medical recordvia a patient's mobile communications device as discussed above, thecomputer readable medium containing instructions that when executed byat least one processor cause the at least one processor to perform amethod. The method may include causing an electronic transmission of atoken to the mobile communication device of the patient for use by amedical image capture application on the mobile communications device. Amedical image capture application may include a program, software orcode that operates on the mobile communication device. The method mayinclude enabling via the medical image capture application, the patientto capture a medical image using a camera of the mobile communicationsdevice. The medical image may include an image of a testing kit, aportion of the testing kit, a skin feature, a biological fluid, areagent, a plurality of reagents, or a combination thereof. The methodmay include enabling via the medical image capture application,processing of the image to generate medical data and to transmit themedical data along with the token, for verifying an identity of thepatient. The method may include receiving the medical data and thetoken. The method may include verifying an identity of the patient usingthe token. Following identity verification, the method may includeautomatically populating an electronic medical record of the patientwith the medical data.

Disclosed embodiments may involve receiving a first image of a pluralityof adjacent wounds in proximity to a form of colorized surface havingcolored reference elements thereon, wherein each wound has multiplesegments of differing colors. By their nature, wounds are not completelyuniform and therefore exhibit segments having differing colors. In someinstances, there are significant differences between wound segments andin some instances color variations of segments may be in the same familyof colors. Colorized elements may be provided on a colorized surface toserve as references for analyzing wound segment colors. The colorizedsurface may take on any form in any shape or size, so long as thesurface serves the intended analysis function. One example of a form ofcolorized surface is illustrated in FIG. 15A, where, on colorizedsurface 1502 a, circular elements 1504 a and 1504 b exhibit differingcolors. The form of colorized surface 1502 a is illustrated as somewhatrectangular, however, this is only an example. It can be of any shape,size or material, so long as it achieves the function of serving as asource of color reference elements for a wound.

When patients have multiple wounds adjacent one another, distinguishingthe wounds from each other and tracking their healing process can posechallenges. Disclosed embodiments enable the tracking of multiple woundsto provide feedback on the healing progress of each wound. To this end,disclosed embodiments may involve a first image of a plurality ofadjacent wounds, which in some embodiments may be in proximity to a formof colorized surface.

Similarly, disclosed embodiments may involve receiving a second image ofthe plurality of wounds, for example in proximity to a form of colorizedsurface, to determine second colors of the plurality of wounds, whereincapture of the second image occurs at least one day after capture of thefirst image. The second image may be similar to the first image, buttaken at least a day later. In the second image, the form of colorizedsurface may be the same instance of colorized surface as appeared in thefirst image; may be a duplicate of the colorized surface used in thefirst image, or may differ from the colorized surface in the firstimage, either in form or substance. In addition, the colorized surfacein the second image may be placed in substantially a same location as inthe first image, or may be placed in a different location. In someexamples, only one of the first image and second image may include adepiction of a form of colorized surface, both the first image andsecond image may include a depiction of a form of colorized surface,both the first image and second image may include no depiction of a formof colorized surface, and so forth. In some examples, any one of thefirst image and second image may include any number of colorizedsurfaces (such as no colorized surfaces, one colorized surface, twocolorized surfaces, three colorized surfaces, more than three colorizedsurfaces, and so forth).

By way of one example, FIGS. 15A and 15B depict sequential images of awound progression together with a color comparison surface. Inparticular, FIG. 15A illustrates image 1500 a captured five days beforethe image of the same wounds in image 1500 b of FIG. 15B. Image 1500 aand/or image 1500 b may have been captured by a device that may have animage sensor (or that may be otherwise associated with an image sensor),such as, for example, mobile communications device 115 or server 145. Insome embodiments, image 1500 a and/or image 1500 b may have beencaptured, sent, received, etc. as part of an image analysis process,such as process 1700, described with respect to FIG. 17. In someexamples, both image 1500 a and image 1500 b may have been captured bythe same device, while in other examples, image 1500 a and image 1500 bmay have been captured by different devices that may have differentcapturing capabilities and/or settings. In some examples, both image1500 a and image 1500 b may have been captured using identical orsubstantially similar capturing parameters, while in other examplesimage 1500 a and image 1500 b may have been captured using differentcapturing parameters (for example, using different exposure time,different shutter speed, different aperture, different ISO, differentlens, different illumination, different viewing angle, different framerate, different resolution, and so forth).

As respectively illustrated in FIGS. 15A and 15B, image 1500 a mayinclude a colorized surface 1502 a and image 1500 b may include acolorized surface 1502 b, either or both of which may be similar to aninstance of colorized surface 132, for example as illustrated in FIG.4A. However, any number of colorized surfaces may be included in animage. Colorized surface 1502 a may depict at least one referenceelement 1504 a and colorized surface 1502 b may depict at least onereference element 1504 b. Colorized surface 1502 a and/or colorizedsurface 1502 b may be configured for placing on human skin. In someexamples, colorized surface 1502 a and/or colorized surface 1502 b mayinclude any number of positioning markers (for example, no positioningmarker, one positioning marker, two positioning markers, threepositioning markers, more than three positioning markers, and so forth),such as positioning marker 410. Such positioning markers may beconfigure to enable detection of a position of a colorized surface byimage analysis, to enable determination of an orientation of a colorizedsurface by image analysis, to enable identification through imageanalysis of portions of the colorized surface (such as referenceelements) that may be based on their position with relation to thepositioning markers, to enable measurement of a curvature of thecolorized surface (for example due to an attachment of the colorizedsurface to the skin) through image analysis, and so forth. For example,colorized surface 1502 a and/or colorized surface 1502 b may be made ofany combination of plastic, paper, paint, temporary tattoo, film, or anyother material displaying a colorized surface. Colorized surface 1502 aand/or colorized surface 1502 b may also include an adhesive material,which may be on a back side of colorized surface 1502 a and/or 1502 b.In some embodiments, colorized surface 1502 a and/or colorized surface1502 b may be made of a flexible material, which may allow the colorizedsurface to conform to contours of a surface (i.e., a portion of skin ofa user) on which the colorized surface is configured to be placed and/oradhered. Colorized surface 1502 a and/or colorized surface 1502 b may beprinted or otherwise generated using any suitable process.

As previously mentioned, the versions of colorized surfaces in the firstimage and the second image may differ. By way of example, in someembodiments, a printed form of colorized surface 1502 a and/or colorizedsurface 1502 b may be of a version that corresponds to a stage ofhealing progress of a wound. Thus, a version of colorized surface 1502 ain image 1500 a may differ from a version of colorized surface 1502 b inimage 1500 b. In some disclosed embodiments, the form of colorizedsurface is a printed form and a same version of the printed form appearsin both the first image and the second image. Thus, a version ofcolorized surface 1502 a in image 1500 a may be the same as a version ofcolorized surface 1502 b in image 1500 b (i.e., the same version of aprinted form may appear in both image 1500 a and image 1500 b). In someembodiments, image 1500 a and/or image 1500 b may not include anycolorized surface.

Reference elements 1504 a and/or 1504 b may be colorized for comparisonto at least one wound. In some embodiments, the number, color, spacing,shape, orientation, placement, etc. of reference elements 1504 a and/or1504 b may be based a characteristic of a user, including acharacteristic of a wound of the user. For example, reference elements1504 a and/or 1504 b may have any combination of a number, color,spacing, shape, orientation, placement, etc. based on a user's woundbeing a burn wound, a bite wound from a particular animal, an abrasion,a laceration, etc. As another example, reference elements 1504 a and/or1504 b may have a number, color, spacing, shape, orientation, placement,etc. based on a user's skin pigmentation, a user's age, a user's gender,a medical condition of a user, etc. In some embodiments, referenceelements 1504 a and/or 1504 b may be based on a stage of healingprogress of a wound. Reference elements 1504 a and/or 1504 b may havespecific chromatic properties associated with them (for example, thechromatic properties discussed regarding FIGS. 1A, 9, and 11).

As shown in FIGS. 15A and 15B, image 1500 a and image 1500 b may includewounds 1506 a, 1506 b, and 1506 c, which may be in proximity to a formof a colorized surface. However, these wounds are merely exemplary, andany number of wounds may be in an image (such as no wound, a singlewound, two wounds, three wounds, more than three wounds, and so forth).Also, while respective wounds in images 1500 a and 1500 b correspondwith each other, embodiments may exist where a wound appears in oneimage, but does not appear in another image (for example, image 1500 amay include a depiction of a wound that do not appear in image 1500 b,image 1500 b may include a depiction of a wound that does not appear inimage 1500 a, both image 1500 a and image 1500 b may include a depictionof the same wound, any combination of the above, and so forth). Thenumber of the wounds in one image (such as image 1500 a) may beidentical to the number of the wounds in another image (such as image1500 b), may be different than the number of the wounds in another image(such as image 1500 b), may be lower than the number of the wounds inanother image (such as image 1500 b), may be higher than the number ofthe wounds in another image (such as image 1500 b), and so forth. Due tothe time lapse between image 1500 a and 1500 b, the appearance of wounds1506 a, 1506 b, and 1506 c may likely differ between the two images, asis illustrated. Healthy human skin may or may not separate one woundfrom another in image 1500 a and/or image 1500 b. In some embodiments,wounds may be of similar or different types, such as a burn wound causedby heat, a chemical burn wound, a bite wound from a particular animal,an abrasion, a laceration, a wound resulting from surgery, etc. Woundsmay also be associated with a particular stage of healing. In someembodiments, a wound may have any number of segments that have differentvisual appearances (e.g., color, color gradient, combination of colors,reflectiveness). These segments may be differentiated and detected by adevice that processes an image of a wound, consistent with the disclosedembodiments.

In some embodiments, image 1500 a and 1500 b may each bear a timestamp1508 a and 1508 b, respectively. The timestamp may be visual, asillustrated, and/or may be contained within metadata. A timestamp maycorrespond to a time and/or date at which an image was captured by animage processing device or may correspond to a time and/or date at whichan image was sent or received by a device. In some embodiments, multipletimestamps may be included with an image (e.g., to indicate a time atwhich an image was captured, when it was sent to a device, when it wasprocessed, when it was viewed, and so forth). A timestamp may beincluded with an image in a variety of ways. For example, a timestampmay be superimposed onto the image itself (e.g., as shown in FIG. 15).In some embodiments, a timestamp may also be embedded in an image.

In some embodiments, image 1500 a and/or image 1500 b may also includemetadata (not shown). Such metadata may include, without limitation, adevice identifier (e.g., based on a MAC address, IP address, portnumber, serial number, etc. of a processing device), user identificationinformation (a name, address, phone number, social security number,insurance number, username, medical test number, etc.), patientinformation, a medical condition, a wound type, information associatedwith a medical professional (e.g., name of a primary care physician orwound specialist), a country of residence of the user, and/or atimestamp.

Disclosed embodiments may involve using the colored reference elementsin the first image to determine first colors of the plurality of wounds,wherein during determination of the first colors, the colored referenceelements may be used to correct for local illumination conditions. Forexample, one or more of the colored reference elements in the firstimage may be compared with image segments containing the wounds, todetermine one or more colors of each wound. More than one coloredreference may be used in order to account for local illuminationconditions. That is, due to shading, non-uniform lighting, glare, or anyother condition, wound colors may be misperceived by the image sensor.Similar misperceptions are likely to occur when processing the coloredreference elements. Since the colored reference elements are known inadvance (or can be determined by comparison to other colored referenceelements), one or more correction factors can be applied to the imagesof the wounds, so that accurate wound colors can be determined, forexample as described above.

Similarly, disclosed embodiments may involve using the colored referenceelements in the second image to determine second colors of the pluralityof wounds, wherein during determination of the second colors, thecolored reference elements may be used to correct for local illuminationconditions. The explanation in the previous paragraph with regard to thefirst image applies equally to the second image.

Disclosed embodiments may involve using the reference elements and/orthe positioning markers in an image (such as the first image and/or thesecond image) to estimate length, size, depth, and/or volume associatedwith a wound. For example, known length and/or size of the referenceelements and/or the positioning markers may be used to estimate adistance from the image sensor, the estimated distance of the referenceelements and/or the positioning markers from the image sensor may beused to estimate the distance of at least part of the wound from theimage sensor, and the estimated distance of at least part of the woundfrom the image sensor may be used to estimate length, size, depth,and/or volume associated with wound based on the length and/or size ofat least a portion of the wound in pixels in the image. For example, itmay be estimated that the distance of the wound from the image sensor isthe same as the distance of the reference elements and/or thepositioning markers from the image sensors. In another example, acurvature of the skin may be estimated, for example based on thepositioning markers and/or on relations between distances of objectswithin the colorized surface, and the curvature may be used togetherwith the estimated distance of the reference elements and/or thepositioning markers from the image sensor to estimate the distance of atleast part of the wound from the image sensor. In some examples, acorrection factor may be calculated based on a relation between theknown length and/or size of objects and the length and/or size of theobjects on the colorized surface in pixels, and the correction factormay be used in order to transform the length and/or size of a portion ofthe wound in pixels in the image to real measurements, for example bymultiplying the length and/or size in pixels by the correction factors.In some examples, a machine learning model may be trained using trainingexamples to estimate length, size, depth, and/or volume of skin features(such as wounds, portions of wounds, etc.) from images of the skinfeatures and colorized surfaces, and the trained machine learning modelmay be used to analyze the image (such as image 1500 a and/or image 1500b) and estimate the length, size, depth, and/or volume of a skinfeature. Such training examples may include an image of a wound with acolorized surface together with an indicator of a measurement of thewound and/or of a particular portion of the wound, a portion of animage, color information associated with an image, and/or any other datacapable of training a machine to estimate the length, size, depth,and/or volume of a skin feature.

By way of one example, FIG. 16 depicts a schematic illustration of awound image analysis processing system 1600, which may be used toprocess an image from a device and process a sequence of instructionsbased on subsequent actions taken at the device. In some embodiments,wound image analysis processing system 1600 may involve the use of amobile communications device 1604 a (e.g., a smartphone) that maycommunicate with communications network 1606, such as by sending image1500 a to communications network 1606. Mobile communications device 1604a may be operated by a user 1602 a. User 1602 a may send image 1500 afrom mobile communications device 1604 a to communications network 1606.In another example, mobile communications device 1604 b may analyzeimage 1500 b to generate image-related information based on image 1500b, and may send the generated image-related information based on image1500 b to communications network 1606. In some embodiments, image 1500 bmay also be sent to communications network 1606, such as by mobilecommunications device 1604 b (e.g., a tablet) which may be operated by auser 1602 b. In another example, mobile communications device 1604 a mayanalyze image 1500 a to generate image-related information based onimage 1500 a, and may send the generated image-related information basedon image 1500 a to communications network 1606. Images 1500 a and 1500 bmay be captured, processed, and/or sent by the same mobilecommunications device, or different mobile communications devices, asillustrated in FIG. 16. Images 1500 a and 1500 b may be captured,processed, and/or sent by the same user, or by different users, asillustrated in FIG. 16. Image 1500 a and/or image 1500 b may includemetadata, such as the metadata discussed with respect to FIG. 15.

Image 1500 a and/or image 1500 b and/or image-related informationreceived at communications network 1606 may be forwarded to a medicalanalysis unit 1608. Medical analysis unit 1608 may include a server1610, which may be coupled to one or more physical or virtual storagedevices such as a database 1612. Server 1610 and/or database 1612 maycontain programs, rules, applications, instructions, etc. used toprocess image 1500 a, image 1500 b and/or image-related information andperform medical analysis on information obtained from image 1500 aand/or image 1500 b. For example, medical analysis unit 1608 may carryout process 1700, described with respect to FIG. 17 below.

FIG. 17 depicts a process 1700 for analyzing an image of a wound, whichmay be performed by any combination of processing devices. Any of theinstructions described herein may include text, an image, an animation,and/or a graphical user interface (GUI). In some embodiments,instructions may be displayed with selectable buttons, sliders, and/orother interactive graphical elements. By way of example only, at anystep in process 1700 (e.g., after an action of one of the steps has beenperformed), a processing device may display a confirmation button, andmay not proceed to another step in process 1700 until a selection inputis detected at the confirmation button.

Disclosed embodiments may include receiving a first image of a pluralityof adjacent wounds, for example in proximity to a form of colorizedsurface having colored reference elements thereon. As discussed, eachwound may have multiple segments of differing colors. Other disclosedembodiments may include receiving a first image of a plurality ofadjacent wounds. In step 1702, a first image and/or image-relatedinformation based on the first image may be received. The first imageand/or the image-related information based on the first image may bereceived at a processor, regardless of where the processor is located.At step 1704, a first set of wounds may be detected. Further, in someexamples, at step 1704 a first colored reference set may be detected.The colored reference set may be any set of color references uses toascertain wound colors. In step 1706, the colors in the first image maybe corrected. Specifically, due to local illumination conditions, theimage sensor may misperceive actual wound colors. However, since thecolored reference elements are captured in the same first image as thewounds and the colored reference elements are of known colors, if thecolored reference elements are misperceived, the extent of themisperception can be determined, and an appropriate correction factormay be applied to wound colors in the first image, for example asdescribed above. In some examples, step 1706 may be excluded fromprocess 1700.

Similarly, in step 1708, a second image and/or image-related informationbased on the second image may be received, at a later time. The latertime may be as short as a day later, and can be multiple days later oreven longer. The second image may be captured by the same or a differentimage sensor that captured the first image. In much the same way as thewound set and colored reference set were detected in the first image, instep 1710 the second wound set and/or colored reference elements may bedetected and in step 1712 the second image may be corrected. In someexamples, step 1712 may be excluded from process 1700.

Next, in step 1714, a comparison may take place to match wounds in thefirst image with wounds in the second image. Due to progression of thewounds (such as healing) or other changes that may have occurred duringthe time lapse from the capture of the first image to capture of thesecond image, the shape, tissue composition, and/or color of one or moreof the wounds may have changed. Therefore, image analysis may uselocation, relative size, distinct features, or other characteristics tomatch wounds in the second image with wounds in the first image. It isto be understood that herein any reference to a healing of a wound (suchas healing progress, level of healing, etc.) may also refer to aworsening in the condition of the wound.

After the wounds are matched in step 1714, in step 1716, detectedchanges from the first wound to the second wound may be used to assessthe wound progression and determine healing progress. By way of exampleonly, one or more of changes in dimensions of the wound, changes incomposition of tissue type within the wound, changes to the peri-woundskin, changes in surface features, changes in color, changes in texture,and/or changes in other characteristics may be used to determine theprogress of wound healing.

While it is to be understood that the steps of process 1700 is notlimited to the specific structures disclosed herein, structuresdisclosed herein (and other alternative structures) may be used toperform process 1700. By way of example, at step 1702, processing device(e.g., a server 1610) may receive an image. This image may be an image1500 a, which may have been captured using an image sensor of a device(e.g., mobile communications device 1604 a). This image may include aplurality of wounds, which may or may not be in proximity to a form ofcolorized surface, as described with respect to FIG. 15.

At step 1704, a processing device, which may be different from theprocessing device receiving the image at step 1702 (e.g., an imageanalysis unit connected to communications network 1606), may detect afirst wound set (e.g., a wound or plurality of wounds) and/or may detecta first colored reference set (e.g., colorized surface 1502 a). In someembodiments, a first wound set and/or a colored reference set may bedetected based on portions of human skin separating wounds or separatinga wound from a colored reference set. A first wound set may also bedetected based on a comparison of the colorized reference elements tochromatic information of the wound set. In other embodiments, such asthose where no form of a colorized surface is in the image, a firstwound set may be detected based on analyzing the chromatic information,contrast with human skin, orientation, shape, etc. of the wound set. Forexample, a processing device may compare the received image to otherimages of wounds stored in a database (e.g., database 1612), and/or mayapply an image analysis algorithm to the received image, where thealgorithm contains parameters related to chromatic properties, size ofregions (e.g., a wound region, skin region, etc.), etc. In someembodiments, the processing device may determine a set of colors,orientation, shape, chromatic information, etc. of the first wound set.In some examples, a machine learning model may be trained using trainingexamples to detect wounds in images, and the trained machine learningmodel may be used to analyze the image and detect the first wound set.Such image may include any number of colorized surfaces, including nocolorized surface, one colorized surface, two colorized surfaces, morethan two colorized surfaces, and so forth. Such training examples mayinclude an image of wounds with no colorized surface together withlabels indicating the wounds in the image and/or the locations of thewounds in the image, a portion of an image, color information associatedwith an image, and/or any other data capable of training a machine todetect wounds and wound information. Another example of such trainingexample may include an image of wounds with one or more colorizedsurfaces, together with labels indicating the wounds in the image and/orthe locations of the wounds in the image. In some examples, anartificial neural network (such as deep neural network, convolutionalneural network, etc.) may be configured (for example, manually, usingmachine learning methods, by combining other artificial neural networks,etc.) to detect wounds in images, and the artificial neural network maybe used to analyze the image and detect the first wound set.

In disclosed embodiments, during determination of the first colors, thecolored reference elements may be used to correct for local illuminationconditions. By way of example, at step 1706, a processing device maycorrect the first image. In some embodiments, the processing device mayuse the first colored reference set in the image to determine localillumination conditions. The processing device may also determinechromatic properties of the first reference set, which it may do bydirectly analyzing the image itself, and/or by examining data containingchromatic property information of the first reference set. This data maybe included within metadata of the image, or may be accessed by readinga machine-readable code (e.g., a scannable code attached to a referenceset, which may be an instance of unique code 906). Based on thedetermined local illumination conditions, chromatic properties, and/orbased on capturing parameters, the processing device may correct theimage. Image correction may physically occur to an image, or it maysimply occur through calculations without a physical alteration of theimage. Some disclosed embodiments may include using the coloredreference elements to determine the local illumination conditions andseparately rectifying colors of the multiple segments of each woundbased on the local illumination conditions. In some examples, step 1706may be excluded from process 1700.

Continuing with the above implementation example, at step 1708, aprocessing device (e.g., a server 1610) may receive a second image. Thisimage may be an image 1500 b, which may have been captured using anyimage sensor, such as a camera associated with the smart phone 1604 a,or, as illustrated, a tablet 1604 b. As with image 1500 a, image 1500 bmay include a plurality of wounds and a colorized surface, as describedwith respect to FIG. 15. In some embodiments, the second image may havebeen captured by an image sensor and/or received at the processingdevice at a threshold amount of time after a first image is capturedand/or received (e.g., the second image is captured at least one dayafter a first image is captured).

Continuing with the implementation example, at step 1710, a processingdevice, which may be different from the processing device receiving theimage at step 1708 (e.g., an image analysis unit connected tocommunications network 1606), may detect a second wound set (e.g., awound or plurality of wounds) and/or a first colored reference set(e.g., colorized surface 1502 b). For example, the second image may beanalyzed and the second set of wounds may be detected in a similarfashion to the techniques described with respect to step 1704, the firstimage and the first set of wounds above. In some embodiments, a secondwound set and/or a colored reference set may be detected based onportions of human skin separating wounds or separating a wound from acolored reference set. A second wound set may also be detected based ona comparison of the colorized reference elements to chromaticinformation of the wound set. In some embodiments, the processing devicemay determine a set of colors, orientation, shape, chromaticinformation, etc. of the second wound set.

Then, at step 1712, a processing device may correct the second image.This correction may be accomplished according to any combination ofactions described above in connection with correction of the firstimage. For example, processing device may determine local illuminationproperties of the image, and may correct the image based on thoseproperties. Continuing with this example, if the light quality causes aspecific misperception of the known color reference elements, then thecorrection necessary for the color reference elements may be applied toa wound, thereby correcting a wound color. Some examples of techniquesfor such correction of the image are described above. In some examples,step 1712 may be excluded from process 1700.

At step 1714, a processing device may match at least one wound in thefirst wound set (e.g., a wound set in image 1500 a) to at least onewound in the second wound set (e.g., a wound set in image 1500 b). Insome examples, a machine learning model may be trained using trainingexamples to match wounds from pairs of wound sets, and the trainedmachine learning model may be used to analyze the first wound set andthe second wound set to match at least one wound in the first wound setto at least one wound in the second wound set. An example of suchtraining example may include a pair of wound sets, together with amatching of wounds between the two wound sets. For example, such woundset may include properties of the wounds, such as dimensions, tissuecomposition, relative position, appearance (for example, an image of thewound), etc., and the machine learning model may analyze theseproperties. In some examples, a machine learning model may be trainedusing training examples to match wounds from pairs of images, and thetrained machine learning model may be used to analyze the first image(such as image 1500 a) and the second image (such as image 1500 b) tomatch at least one wound in the first image to at least one wound in thesecond image. Such training examples may include a pair of images,together with a matching of wounds between the two images, a portion ofan image or portions of multiple images, color information associatedwith an image, and/or any other data capable of training a machine tomatch wounds.

In some embodiments, matching the wounds may include predicting anexpected appearance of a wound (e.g., predicting an expected color,size, etc. of a wound in an image taken later in time than a firstimage). Disclosed embodiments may include determining a time differencebetween the first image and the second image. For example, predicting anexpected appearance of a wound may involve determining a time differencebetween a capture time of a first image and a capture time of a secondimage, which may be accomplished by determining a time lapse betweenimage capture. For example, a processing device may determine when animage was captured by reading a timestamp 1508 a, and/or timestamp 1508b. In embodiments where a timestamp is superimposed on an image, theprocessing device may use optical character recognition to read thetimestamp. In other embodiments, such as when a timestamp is embeddedinto an image or attached to it within metadata, the processing devicemay extract it from those sources. In some disclosed embodiments, thetime difference between the first image and the second image may bedetermined automatically using metadata associated with the secondimage. For example, a processing device may determine the timedifference automatically using metadata associated with the secondimage. In some disclosed embodiments, the time difference between thefirst image and the second image may be determined automatically bycomparing metadata associated with the first image and metadataassociated with the second image. This may be accomplished by aprocessing device.

Based on a time difference between images of a wound set, a processingdevice may determine an expected appearance of at least one wound. Forexample, data may be maintained in a data structure that maps a healingprocess of a wound based on wound characteristics. Alternatively,learning algorithms may be applied to a repository of wound images toidentify wounds that most closely correspond to the first image, andthereby predict how the current wound is expected to heal over time.Since the time lapse is known for the first and second image, based onhow other similar wounds of others have healed over time, the system candetermine if the wound healing is progressing as expected, or if thereappears to be an abnormality. Such progress may be based on anycombination of a change in color of the wound, a reduction in size ofthe wound, a change in the shape of the wound, a change in the color ofan outline of the wound, a change in the tissue composition of thewound, and/or non-wound-related characteristics, such as a patient'sage, gender, health, genetics, skin type, or any other non-wound-relatedcharacteristic that might correlate to wound healing.

Disclosed embodiments may include predicting an expected appearance ofeach of the plurality of wounds in the second image based on thedetermined time difference and/or using the predicted expectedappearance for matching each of the plurality of wounds in the secondimage to the plurality of wounds in the first image. By way of example,after a processing device has predicted the expected appearance of awound in a second image, the processing device may use the predictedexpected appearance to match at least one wound in a first image to atleast one wound in the second image (e.g., matching each of a pluralityof wounds in the second image to wounds of a plurality in the firstimage). In disclosed embodiments, the predicted expected appearance maybe based on a type of each of the plurality of wounds. For example, alaceration is different in type from a burn, and therefore, the healingprocess for the wounds would be expected to be different. Indeed, thereare many different types of wounds, ranging from chemical burns,sunburns, lacerations, abrasions, contusions, hematomas, punctures, andavulsions. Each has its own wound healing profile.

In some embodiments, the predicted expected appearance may be based notonly on the type of a wound but also on its extent. For example, largeror deeper wounds would be expected to have a different healing processthan small or shallower wounds.

In some cases, a wound may split into two or more wounds over time, andtwo or more wounds may be joint together into one wound over time. Insome embodiments, a processing device may match two or more wounds inthe first wound set (e.g., a wound set in image 1500 a) to a singlewound in the second wound set (e.g., a wound set in image 1500 b),and/or may match a single wound in the first wound set (e.g., a woundset in image 1500 a) to a plurality of wounds in the second wound set(e.g., a wound set in image 1500 b). For example, two or more wounds inthe first image (e.g. in image 1500 a) may be joined together into afirst wound in the second image (e.g. in image 1500 b), and step 1714may include matching the first wound in the second image to the two ormore wounds in the first image. In another example, a first wound in thefirst image (e.g. in image 1500 a) may split into two or more wounds inthe second image (e.g. in image 1500 b), and step 1714 may includematching the two or more wounds in the second image to the first woundin the first image. In some examples, a machine learning model may betrained using training examples to match a plurality of wounds in a onewound set with a single wound in another wound set, and the trainedmachine learning model may be used to analyze the first wound set andthe second wound set to match a plurality of wounds in a first wound setwith a single wound in a second wound set and/or to match a single woundin a first wound set with a plurality of wounds in a second wound set.Such training examples may include a pair of wound sets together with amatching of a plurality of wounds in one wound set with a single woundin the other wound set, a portion of an image or portions of multipleimages, color information associated with an image, and/or any otherdata capable of training a machine to estimate. For example, such woundset may include properties of the wounds, such as dimensions, tissuecomposition, relative position, appearance (for example, an image of thewound), etc., and the machine learning model may analyze theseproperties. In some examples, a machine learning model may be trainedusing training examples to match a plurality of wounds in a one imagewith a single wound in another image, and the trained machine learningmodel may be used to analyze the first image (such as image 1500 a) andthe second image (such as image 1500 b) to match a plurality of woundsin a first image with a single wound in a second image and/or to match asingle wound in a first image with a plurality of wounds in a secondimage. An example of such training example may include a pair of images,together with a matching of a plurality of wounds in one image with asingle wound in the other image.

Disclosed embodiments may include determining an indicator of thehealing progress for each of the plurality of wounds based on changesbetween the first image and the second image. An indicator may be anydisplayed measure of wound progress, whether represented graphically,pictorially, textually, or otherwise. An indicator may provide anotification that a wound is healing faster or slower than expected; mayadvise of an expected time remaining until substantial healing iscomplete; may provide a pictorial or video time lapse indication ofhealing progress; may provide a warning of slow progress or regressionor may include any other indicator that provides a sense of healingprogress. In some embodiments, the predicted expected appearance may bebased on image comparison with a repository of wound images, and ahealing progress indicator may be provided for each of the plurality ofwounds determined from previous images. In some embodiments, a healingprogress indicator may also be used to update medical records. Forexample, a processing device may update personal electronic medicalrecords with at least one healing progress indicator for a wound, whichmay be part of a plurality of wounds. Some disclosed embodiments mayinclude updating personal electronic medical records with the indicatorof the healing progress for each of the plurality of wounds.

In some embodiments, a processing device may use wound signatures tomatch a wound in one set to a wound in another set. A wound signaturemay include any unique characteristic that may distinguish one woundfrom another. Disclosed embodiments may include determining a woundsignature based on visual appearance of the multiple segments for eachof the plurality of wounds. In some embodiments, at least one woundsignature may be based on at least one visual appearance of a segment ofa wound. In yet other embodiments, multiple visual appearances ofmultiple segments of a wound may be used. In other words, a woundsignature may encompass a combination of segments of a wound thatuniquely differentiates it from other wounds (either uniquely from allwounds in the same image, or uniquely from all possible wounds). Forexample, a wound may have a particular combination of segments havingdifferent appearances (based on colors, shapes, sizes, tissue type,etc.) Disclosed embodiments may include using the wound signature formatching each of the plurality of wounds in the second image to the eachof the plurality of wounds in the first image. For example, acombination of segments may form a wound signature, which a processingdevice may use to match each of a plurality of wounds in a second imageto the each of the plurality of wounds in a first image. For example, awound signature may be determined based on color distribution ofmultiple segments for each of a plurality of wounds. In someembodiments, a wound signature may be associated with ratios betweenareas of the multiple segments. In some embodiments, an auto-encoder maybe used to generate wound signature of an image of a wound. Suchauto-encoder may include a deep neural network trained using images ofwounds.

Wound signatures may also be updated over time, which may improveaccuracy of wound-matching. Disclosed embodiments may include updatingthe wound signature for each of the plurality of wounds based on visualappearance of the multiple segments as depicted in the second image. Forexample, in some embodiments, a wound signature from a first image maybe updated for at least one wound based on at least one visualappearance of at least one segment of a wound as depicted in a secondimage. While two images are discussed in this example, any number ofimages may be used to create accurate wound signatures. Moreover, whileportions of the description refer to a first image and a second image,this disclosure contemplates that additional images may be captured andused to track wound healing progress over time.

At step 1716, a processing device may determine healing progress of atleast one wound, which may be based on a change in appearance of the atleast one wound between two images. For example, after matching onewound of a plurality of wounds in a first image to a wound in aplurality of wounds in a second image, the processing device maydetermine that a combination of color, dimensions, shape, tissuecombination, condition of the peri-wound skin, etc., has changed withrespect to the wound. These changes may be associated with a particularlevel of healing progress. For example, a processing device maydetermine that a size of a matched wound has shrunk and/or that a hue,value, and/or intensity of at least a segment of the wound has changed.Based on that determination, the processing device may establish thatthe wound has healing to a certain point of healing progression (75%fully healed, healing faster than expected, wound is infected, healinghas stagnated, etc.) In some examples, a machine learning model may betrained using training examples to determine levels of healingprogresses from pairs of images of a wound, and the machine learningmodel may be used to analyze the depiction of the wound in the firstimage and the depiction of the wound in the second image to determinethe level of healing progress of the wound. An example of such trainingexample may include a pair of images of a wound, together with a labelindicating the level of healing progress of the wound. In some examples,a machine learning model may be trained using training examples todetermine levels of healing progresses from pairs of image-relatedinformation records, and the trained machine learning model may be usedto analyze image-related information based on the first image (such asimage 1500 a) and image-related information based on the second image(such as image 1500 b) determine the level of healing progress of thewound. Such training examples may include a pair of image-relatedinformation records, together with a label indicating the level ofhealing progress of the wound, a portion of an image, color informationassociated with an image, and/or any other data capable of training amachine to determine a level of healing progress of a wound.

Disclosed embodiments may include using the first captured image, thesecond captured image, and additional captured images to create a videostream illustrating the healing progress for each of the plurality ofwounds. By way of example, a processing device may generate a videoillustrating healing progress of at least one wound. A first image of awound, a second image of a wound, and possibly additional images, may beused to generate the video. Some embodiments may include generatingartificial images as frames between the captured image frames in thegenerated video. These artificial images may be estimates of how thewound changed between capture of the first image and capture of thesecond image. The artificial images may enable a video playback wherethe first image morphs into the second image. Some embodiments mayinclude using the colorized surface in the captured images to adjust atleast one of orientation of the plurality of wounds and colors of theplurality of wounds in the video. For example, a processing device mayuse the chromatic information of the colorized surface to determinechromatic information of a plurality of wounds, which it may use todetermine relative orientations and/or positions of wounds, which it mayin turn use to adjust an orientation of a wound, or, in some cases, anorientation of the entire image. In addition, misperceptions of color byimage capture device due to local lighting conditions, can be correctedin either video playback or in still images. Thus, a patient's medicalrecord may include color-corrected video or images.

Disclosed embodiments may include determining that the healing progressof at least one of the plurality of wounds is below a healing thresholdand/or generating a treatment suggestion for improving healing of the atleast one wound. By way of example, a healing threshold may comprise acombination of chromatic properties, shape, size, and/or otherattributes expected for at least one segment of a wound.

Based on the determination of the healing progress being below athreshold, the processing device may generate and/or provide recommendedtreatment steps (i.e., suggestions) to a user (e.g., user 1602 a, user1602 b, etc.), who may be the patient or who may be a healthcareprofessional. For example, the processing device may recommend theapplication of a particular antibiotic to a wound, reduction of exposureof a wound to sunlight, increasing the amount of a particular nutrientin a user's diet, etc.

Aspects of this disclosure may relate to a method for updating anelectronic medical record based on patient generated image data. Someaspects of such a method may occur electronically over a network that iseither wired, wireless, or both. Other aspects of such a method mayoccur using non-electronic means. In a broadest sense, the method is notlimited to particular physical and/or electronic instrumentalities, butrather may be accomplished using may differing instrumentalities.

Consistent with disclosed embodiments, a method may involve physicallyproviding to a patient a test kit. A test kit, as used herein, mayinclude any materials configured to determining a characteristic of apatient. For example, a test kit may contain material to enable one ormore of monitoring, measuring, analyzing, or capturing of informationreflective of a characteristic of a patient. The test kit may bephysically provided to the patient. Physical provisioning of the testkit may occur in many differing ways. For example, a healthcare provideror an agent of a healthcare provider might transmit the test kit to thepatient by post, private carrier, or by physically delivering the testkit to a location of the patient, be it at home, office, hospital, orany other location where the patient may be located. Alternatively, thetest kit may be provided by inviting the patient to a location where thetest kit may be picked up, or by providing the test kit to the patientduring a scheduled or unscheduled appointment, such as with a doctor, ata clinic, or in any other facility where the patient may be located.Alternatively, providing the test kit to the patient may involve the useof an intermediary, such as a caregiver or agent of the patient (i.e.,providing the test kit to an intermediary, constitutes providing thetest kit to the patient, consistent with usage herein), a deliveryservice, and so forth. Thus, for purposes of this disclosure, a test kitis provided to a patient if the test kit is conveyed directly to thepatient, or if it is provided to an agent of the patient, to enable thetest kit to be used with the patient.

While a test kit may be provided in a single package or otherconveyance, the test kit may also be provided in a hybrid manner in thatelements of the kit may be provided at different times and/or throughdiffering channels. For example, an electronic element of a kit may betransmitted electronically to the patient while a physical component maybe transmitted physically. The manner of transmission may not becritical, so long as the patient is enabled to receive the test kit.

Some aspects of this disclosure may involve sending to the patient averification code. A verification code may be physical and/orelectronic, and may be any mechanism that enables a receiver to verifythat received information originates from an intended source. Althoughreferred to in the singular, a verification code may have multiplecomponents, and may include more than one step, form or level ofauthorization. The code might be visual, in the form of a bar code, a QRcode, or any other graphical form of code. The code may also bealphanumeric. Alternatively, the code could be fully electronic toenable verification between multiple pieces of electronics withoutdirect user involvement. Thus, the code may be contained within aphysical package sent to the patient, or could be electronicallytransmitted to and/or between a device associated with the patient andanother entity.

The verification code may be sent to a patient via any communicationchannel such as an email, text or SMS message, phone call, mail, visualmessage, audible message, or any other communication channel. Theverification code may be electronically transmitted to the mobilecommunications device. The transmission of the verification code maytake place over any wired or wireless communication channel. Forinstance, the code may be transmitted over a PAN, LAN, WLAN, CAN, MAN,WAN, SAN, POLAN, EPN, VPN, cellular network, or any combination thereofor any other network capable of information transmission.

If physically provided with the test kit, the verification code may beprinted on the kit packaging or within the kit, or may be indirectlyprovided in the kit, such as by providing an address of an electroniclink through which the verification code may be obtained orelectronically accessed. Thus, the code may be included with writteninstructions of the kit, as an image within the kit, on a separate itemin the kit, or through any other electronic or physical means. Theverification code may be machine readable.

Embodiments of the disclosure may also involve providing instructions tothe patient to access, via a mobile communications device, anapplication for using the mobile communications device to capture amedical image. Instructions may be provided through one or more ofprinted instructions in the kit, by providing a link to the patient (oran agent of the patient) to electronically access the instructions, byproviding the instructions through an app resident on a mobilecommunications device on the patient side, or on a server side, remotefrom the patient side, or through any other medium or channel thatprovides instructions for conducting a test. Again, reference to accessby the patient alternatively includes providing access to one or more ofthe patient or an agent (e.g., assistant or administrator) associatedwith the patient.

As previously discussed, the mobile communications device can be anydevice with communications capabilities that enables transmission ofimage-related information. The mobile communications device may beenabled to capture a medical image. A medical image can be any imageused for medical purposes. For example, it can be an image of a reactionto a particular test, a biological response, an anatomical image, or animage of a wound or a condition. Other examples of medical images aredescribed above. A device may be enabled to capture a medical image ifit includes an integrated image sensor, or if it has the capability topair with a device that includes an image sensor. For example, a mobilephone or tablet may integrate an image sensor with a transmissioncomponent. Alternatively, a non-networked camera paired with a PC orother networked device may serve as a mobile communications device. Inanother example, a wearable device may include a camera and atransmission component.

Embodiments consistent with this disclosure may also involve enablinganalysis of image-related data to determine an insufficiency of theimage for medical examination. Enabling analysis may involve providingan app to the mobile communications device to perform image analysis onthe mobile communications itself. Alternatively or in addition, themedical image may be transmitted via a network to a remote locationwhere the analysis may take place. Thus, enabling analysis may involveone or more of providing an app that examines the medical image,enabling transmission of the medical image to a location where theexamination may take place, or undertaking the medical imageexamination. The image-related data for examination may include theimage itself, a subset of the image, or data derived from the image.Data derived from the image may include, for example, information aboutlocal illumination conditions, contrast, clarity, subject matter, or anyother image characteristic or image-related information that may enablea determination of whether the image is sufficient for its intendedpurpose. If the image-related data is determined to be inadequate forits intended purpose, it may be deemed “insufficient.” For example, amachine learning model may be trained using training examples todetermine whether image-related data is sufficient for a particularpurpose, and the trained machine learning model may be used to analyzethe image-related data to determine whether it is sufficient for theintended purpose. An example of such training example may includeimage-related together with a label indicating whether the image-relateddata is sufficient for a particular purpose.

The extent and type of analysis may depend on a nature of an associatedtest. At a minimum, the analysis may be used to determine whether theimage is sufficient for accomplishing the purposes of the particulartest (e.g., determining whether the image contains sufficientinformation to enable meaningful medical examination).

Embodiments consistent with the disclosure may involve enabling displayon the mobile communications device of a message indicative of theinsufficiency of the image. Specifically, if the image fails to meet anacceptable standard for medical analysis, a message may be displayed,via the mobile communications device indicating the insufficiency. Themessage may be visual and/or audible. For example, the message maysimply indicate that the image was insufficient, and/or it may direct auser to retake the image. As used herein, a message may be considereddisplayed on a device if the message is either visually or audiblypresented via the device.

Embodiments of the disclosure may also involve, in response to thedisplay of the message indicative of the insufficiency of the image,receiving a new image from the patient. The new image may be an attemptby the patient or an agent of the patient to provide an image that is ofa level acceptable for medical examination. As with the original image,the new image may contain all or part of a captured image, or maycontain data characterizing the actual captured image, each of which isconsidered a receipt of a new image, for purposes of this disclosure.The new image may be received in the same or different manner from theoriginal image. If the analysis occurs locally, the image, orinformation characterizing the image may be received by a localprocessor. If the analysis occurs remotely, the new image or informationcharacterizing the new image may be received via a network transmission.

Embodiments consistent with this disclosure may involve receiving theverification code from the patient. The verification code, in any of theforms described earlier, may be received by an app locally associatedwith a mobile communications device or may be received remotely on aserver or other equipment. Regardless of how or where received, theverification code may serve to confirm that the new image originatedfrom an intended source (e.g., the patient or an agent associated withthe patient). In this way, the verification code may be used to verifythat the new image was received from the patient.

Consistent with this disclosure, embodiments may involve, for exampleupon verification, automatically sending a notification to a healthcareprovider for updating an electronic medical record of the patient withdata associated with the new image. The notification may include one ormore of a computer readable message or a machine readable message. Forexample, the notification may include computer readable patientidentifying information together with either the new image itself, aportion of the image itself, data characterizing the new image, and/ordata based on an analysis of the new image (any and all of which isconsidered a new image for purposes of this disclosure). Thenotification may be in a form that enables a recipient system toautomatically update the patient's medical record with the new image, orwhich enables the record to be updated with limited human intervention(e.g., a human verification prior to update.) Alternatively, thenotification may include human readable data, identifying the patientand enabling a human (such as an agent of a healthcare provider orinsurance company) to update an electronic medical record. In anotherexample, the notification may be sent to a healthcare provider togetherwith information based on the verification code, and the verificationmay occur at the healthcare provider using the received informationbased on the verification code. Further, after the healthcare providersuccessfully verified (using the received information based on theverification code) that the notification includes information receivedfrom the patient, the healthcare provider may update an electronicmedical record of the patient based on the received data.

While the features previously described are not limited to a particularstructure or additional steps, by way of example only, FIG. 21 is aschematic illustration of a method for updating an electronic medicalrecord (EMR) based on patient generated image data. A list of eligiblepatients 2101 and details of the eligible patients may be uploaded intoa patient portal 2102. A local modality team 2103 may be charged with atask of confirming that patient consent is received 2104, in accordancewith either policy or law. In some instances, policy or law may notrequire consent, and if not, this step may be omitted. Next, a medicaltest kit may be delivered to the patient, as represented by node 2105.Simultaneously, before, or after, a verification code may be sent to thepatient. The verification code may take the form of one or more of atext, an SMS message 2106, a token 2107, or any other electronic orphysical mechanism for authenticating a transmission. The patient maydownload an application onto their corresponding mobile communicationsdevice 2108. As previously described, the mobile communications devicemay take the form of a phone, mobile phone, smart phone, smart watch,smart glasses, tablet, laptop, personal computer, PDA, or any device orcombination of devices that may allow for communication between apatient and the system (such as mobile communications device 115, mobilecommunications device 1604 a, mobile communications device 1803, mobilecommunications device 2200, and so forth). The application may guide apatient through the medical testing procedure.

The medical test kit delivered to the patient may be configured formonitoring, measuring, analyzing, or capturing information reflective ofa characteristic of a patient. For example, if used for chemicallyanalyzing a biological material, the test kit may include a containerconfigured to contain, for example, a biological fluid; a dipstickincluding a plurality of test reagent pads thereon for measuringdiffering properties of the biological fluid; and a colorized surfaceincluding a dipstick placement region in proximity to a plurality ofcolored reference elements. For example, a medical test kit may includea dipstick 2109. The dipstick 2109 may include one or a plurality ofreagent pads 2110. The one or plurality of reagent pads 2110 may beconfigured to change color when in contact with a specific analyte,characteristic or parameter. The reagent pads 2110 may include, forexample, a plurality of different reagents to react with differentanalytes, components, characteristics, parameters, etc. found withinbiological fluid. Biological fluid may include sweat, urine, blood,stool, breast milk, saliva, interstitial fluid, or any other biologicalfluid or sample. The medical test kit may also include a container orreceptacle 2111 and colorized surface 2113. The colorized surface 2113may include a dipstick placement region in proximity to a plurality ofcolored reference elements 2114. The container or receptacle 2111 may beof any size, shape, or construction suitable for receiving or containinga desired biological fluid 2112. The medical test kit may also includeadditional components to aid in the collection of a biological fluid.For example, an additional component may include a needle, lancet, orother means to allow for the piercing or puncturing of a skin surface inorder to collect a blood sample. One example of such medical test kitmay include urinalysis kit 1200 described above.

Test reagent pads 2110 may be configured for measuring differingproperties of the biological fluid. For example, if associated with aurine test, the dipstick may include a first test reagent pad formeasuring a first property associated with albumin concentration and asecond test reagent pad for measuring a second property associated withcreatinine concentration. Other examples of dipsticks and/or reagentpads are described above. A plurality of reagent pads may be configuredto measure a plurality of different analytes, concentrations,components, compounds, etc. as desired. Moreover, a first reagent padmay be configured to react with a first biological fluid, and a secondreagent pad may be configured to react with a second biological fluid.The dipstick may include at least two of: a first test reagent pad formeasuring a first property associated with leucocytes in urine, a secondtest reagent pad for measuring a second property associated with bloodin urine, a third test reagent pad for measuring a third propertyassociated with nitrite in urine, a fourth test reagent pad formeasuring a property associated with a urinary tract infection, and afifth test reagent pad for measuring a property associated with whiteblood cells in urine. It is understood that the test kit may include anynumber of test reagent pads 2110 and may be configured for measuring aplurality of different analytes, parameters, or characteristics of abiological fluid, or a plurality of biological fluids.

The application associated with the mobile communications device 2108may guide the patient through the medical test procedure. One example ofsuch application may include app 1804. In other example, suchapplication may implement all or parts of process 700, of process 800,of process 1400, of process 1700, of process 2000, and so forth. In someexamples, the application may prompt a patient to fill receptacle 2111with a biological fluid 2112 such as urine. It is noted however, thatother biological fluids may be used with various medical test kits. Forexample, test kits may be designed for measuring an analyte,concentration, property, parameter, characteristic or other featureassociated with blood, urine, saliva, sweat, stool, interstitial fluidor any other biological fluid or sample. The application may then directthe patient to blot the dipstick 2109 and place it adjacent to colorizedsurface 2113. Thereafter, the instructions may guide the user to usemobile communications device 2108 to capture an image of the dipstick2109 and colorized surface 2113. An analysis of image-related dataassociated with the image may be performed to determine insufficiency ofthe image for medical examination. For example, the image may notinclude the entire colorized surface 2113 or entire dipstick 2109, theimage may include a glare or other local lighting deficiency renderingsome or all of the image obscured, blurry, lacking sufficient contrast,or otherwise failing to capture sufficient information to enablecompletion of the test. In the event that a determination has been madeof insufficiency, the display on the mobile communications device maydisplay a message indicative of the insufficiency of the image. The usermay then be instructed to recapture the image (e.g., provide a newimage).

When a sufficient image has been captured by the mobile communicationsdevice 2108, the image and corresponding image-related data may beuploaded to the server 2115. The image-related data may include at leastone of image clarity, image illumination, and a time of capture of themedical image. Image-related data may also include geographicinformation, weather information, pixel information, camera or imagesensor information, an analysis of the image or a portion thereof,compression data, data related to transmitting the image, or any otherdata or information related to the medical image or capturing of themedical image. Server 2115 may include software and/or hardware to allowfor analysis of the image or image-related data to generate a testresult. The test result may then be then transmitted to server 2116.Servers 2115 and 2116 may be one or more of a plurality of remote orlocal servers, or one or more of a plurality of cloud-based servers.Server 2116 may be associated with a healthcare provider or healthcareprofessional. Server 2116 may also include an electronic medical recordfor the patient which may be updated to include the test result. Theupdated electronic medical record 2117 may be displayed for healthcareprofessional 2118. Healthcare professional 2118 may review medicalrecord 2117 and test result. After review, the healthcare professional2118 may issue a follow up test 2119.

FIG. 22A shows an example mobile communications device 2200. Mobilecommunications device 2200 may include an application to help guide thepatient through the medical testing procedure. Some examples of mobilecommunications device 2200 may include any one of mobile communicationsdevice 115, mobile communications device 1604 a, mobile communicationsdevice 1803, a smartphone, a tablet, a wearable computer, and so forth.As illustrated, on display 2201 of mobile communications device 2200, anintroductory message may be provided to the patient. To begin a medicaltesting procedure, the patient may be prompted to press the “Let'sStart” button 2202. As illustrated in FIG. 22B, the mobilecommunications device 2200 may prompt the patient to enter averification code in field 2212. This ensures for secure communicationbetween a specific patient, or agent of the patient, and one or more endusers. Alternatively, the application could instruct the user to scan acode provided in the test kit or separately sent to the user.

By way of an example, as shown in FIG. 22C, the application may causeinstructions to appear on display 2221 for undertaking a medical test.For example, FIG. 22C may contain graphics illustrating how to place adipstick on the color-board or colorized surface prior to obtaining animage thereof. Upon successful completion of the testing procedure,display 2231 in FIG. 22D may indicate that the test results have beensuccessfully uploaded to the electronic medical record. The forgoinginstruction examples are for illustration purposes only. Step-by-stepinstructions with many more illustrations, diagrams, animations, imagesor audible instructions may be provided to carefully guide the userthrough a testing process.

FIG. 23 illustrates one exemplary method 2300 for updating an electronicmedical record based on patient generated data. The method may involvephysically providing to a patient a test kit at step 2301. Physicallyproviding a test kit may include any means for allowing a patient toaccess a test kit, for example as described above. For example, apatient may receive a test kit from a healthcare provider's office,through mail or a delivery service, from a retail location, or any othermeans allowing for collection of or access to a test kit. Test kits maybe delivered to a specific patient at recurring intervals, wheninstructed by a healthcare provider or insurance provider, or whenrequested by the patient. Additionally, test kits may be delivered toone or more patients as the result of a health preventative program.

Once a patient is in custody of a test kit, instructions may be providedto the patient to capture a medical image. An application associatedwith a mobile communications device may assist the patient in capturingthe medical image of, for example, medical test results. Some examplesof such instructions are described above. In some examples, theinstructions may include guidance on how to perform the medical testusing materials provided in the test kit. This may occur through anapplication running on the mobile communications device or otherwiseaccessible through the mobile communications device that may providedirections, instructions or steps of performing the medical test. Thedirections, instructions or steps may be visually presented to thepatient in the form of text, images, videos, or animations.Additionally, the directions, instructions or steps may be presentedaudibly to a patient in the form of speech, sound bites, or otherprompts. Additionally, the application may include an interactivefeature which may respond to a prompt or question of the patient. Theinstructions provided to the patient may include one or more, or atleast two of opening the test kit, expanding a collapsed measuring cup,dipping a dipstick, blotting the dipstick, placing the dipstick on acolorized test surface; capturing an image of the dipstick on thecolorized test surface, and recapturing the image of the dipstick on thecolorized test surface. The instructions provided to the patient may besequential and the instructions may include directing a user to completeat least one activity during a predefined time window after completionof a previous activity. For example, the instructions may direct thepatient to place the dipstick in the biological fluid for a specifiedtime period (such as a time period between 50 to 65 seconds), and maythen instruct the patient that an image of the dipstick needs to becaptured at a specified time window (for example, between 2 minutes to 3minutes after the dipstick is placed on a colorized test surface,between 2 minutes to 3 minutes after the dipstick is dipped, between 2minutes to 3 minutes after the dipstick is blotted, and so forth).

The method may further include sending to the patient a verificationcode, as depicted in step 2302 of FIG. 23. The order and manner of codetransmission may be insignificant in that it could be physicallytransmitted to the patient with the test kit or transmittedelectronically to the patient before, after, or simultaneously withphysical transmission of the test kit. The verification code may includemore than one step or level of authorization, and may take a pluralityof forms. As previously discussed, so long as the verification code isable to confirm that transmitted medical information is authentic (e.g.,originates from the intended source), the verification code may take anyform and may be received in any manner. The verification code may alsobe used to verify that test kit usage does not exceed a prescribed usagelimit. For example, to avoid a reading of contaminated results orresults distorted by a test kit's exposure to prior biologicalmaterials, each test kit may be designed for limited use (e.g., singleuse), and the verification code may be used to ensure that no test kitus used beyond its prescribed limit. In such situations, eachverification code may be unique to a particular instance of a test kit.

The method of FIG. 23 may include providing instructions to the patientto access, via a mobile communications device, an application for usingthe mobile communications device to capture a medical image. Theinstructions may be included as part of the test kit, or may betransmitted to the patient apart from the test kit. The application maydirect the user and guide the user through the process of capturing amedical image. For example, the instruction may guide the patient incapturing a medical image at step 2303 using a mobile communicationsdevice with an integrated image sensor, or which is otherwise associatedwith an image sensor. In one aspect, the medical image may include animage of a dipstick and corresponding reagents proximate or adjacent toa surface containing colorized reference elements. In another aspect,the image may include one or more skin features proximate to or near acolorized surface. The medical image may be uploaded or transmitted toan analysis unit. In one aspect, the analysis unit may be a remoteserver or processor that analyses the medial image and/or image-relateddata to determine, in a first instance, sufficiency of the image. Inanother aspect, the analysis unit may be a processor or group ofprocessors located within the mobile communications device. Thus,enabling an analysis of image related data may include providing anapplication for use by a mobile communications device, or it may involveproviding the analysis function through the provision of softwarerunning on a remote server. Enabling of analysis may alternatively beaccomplished by providing any other structure or functionality where theanalysis may occur.

The analysis of image-related data may include determining aninsufficiency of the image for medical examination. At step 2304analysis of image-related data may be enabled. The analysis may includeanalyzing the image or a portion of the image, comparing the image or aportion of the image to a threshold, or any calculation or measurementthat can be used to determine if an image is suitable for medicalexamination. In one aspect, enabling analysis includes providing thepatient with access to the application, either by providing software tothe mobile communications device or by enabling the mobilecommunications device to communicate with a remote server where theanalysis occurs. Some examples of an analysis of image-related data aredescribed above.

In step 2305, an insufficiency of the image may be determined for one ormore of the reasons previously discussed. Thereafter, the method mayenable display of an associated message to the user. The message mayinclude one, or a plurality of messages in one or more of text,graphics, animation, audio, video, etc. The message indicative of theinsufficiency of the image may include an explanation of why the imageis insufficient and may point out specific features that render theimage insufficient. For example, the message may state that the entirecolorized surface is not shown, or that the image is too blurry.Alternatively, the message may simply instruct the user to retake theimage. Enabling display of the message indicative of the insufficiencyof the image may include one or more of sending data to a display inorder to cause the message to appear, or may simply involve providingthe application that enables such functionality. Enabling display of themessage indicative of the insufficiency of the image may also includetransmitting data associated with the message to the mobilecommunications device over a wireless network. Displaying the messagemay include any visual or audio display. After displaying the messageindicative of the insufficiency of the image, modified instructions forcapturing the new image may be provided, wherein the instructions may bemodified to account for the cause of prior insufficiency of the image.For example, the application may prompt a user to move the dipstick to anew position before capturing a new image, alter the field of view tocapture the entire colorized surface, or alter a lighting condition.Additionally, the application may also request the user hold the imagesensor steady while capturing an image, or prompt the user to performother tasks to ensure collection of an image and image-data suitable foranalysis.

In step 2306, an exemplary method may include, in response to thedisplay of the message indicative of the insufficiency of the image,receiving a new image from the patient. The new image may similarly beanalyzed for sufficiency in much the same way as the original image wasanalyzed.

At one or more times during the process of FIG. 23, the method mayinclude receiving the verification code from the patient, and using theverification code to verify that the new image was received from thepatient. For simplicity of discussion, verification code receipt isillustrated once at step 2307. However, the verification may instead (oradditionally) have been conducted earlier in the process, essentiallyverifying the entire session or appropriate portions thereof. Afterreceiving the verification code, it may be used to verify that the newimage was received from the patient. This may occur, for example, bytransmitting the code to a remote server that looks up the code andconfirms that the code is in fact assigned to the associated patient.

With some medical tests, a time lapse between conducting of the test andimage capture may be material. If so, and if the image capture failed tooccur within a prescribed time window regulated by the application, aprompt may be displayed to a patient requesting a new image. In oneexample, the user may be instructed to rerun the complete test beforecapturing the new image. In other instances, the user may be instructedto resubmit or retransmit previously captured images. An entirely newimage may be required when the insufficiency is the result of a portionof the test kit missing from the image, or the image being too noisy orblurry. However, in the event where the image was complete but did nottransfer entirely, a resubmission of the same image may be sufficient.

If the image is complete and sufficient, the method may includeanalyzing the new image, wherein the new image depicts a colorizedsurface in proximity to a medical analysis region, and the colorizedsurface enables determining of colors of the medical analysis region.For example, the new image may be analyzed as described above, forexample in relation to process 700 and/or process 800 and/or process1700.

Upon verification that a new image has been received from a patient, anotification may be automatically sent to a healthcare provider in step2309 for updating an electronic medical record of the patient with dataassociated with the new image. If the healthcare provider happens to bethe entity who received the new image in the first instance, and thehealthcare provider also maintains the EMR of the particular patient,then sending the notification may occur internally. If the patient's EMRis maintained by a healthcare provider other than the entity whoinitially received the updated image, the notification may be sent via anetwork or any other suitable means to the EMR-maintaining healthcareprovider for updating the patient's EMR. The notification may take theform of a notice advising the healthcare provider that new results areavailable upon request, or the notification may include indicia of theresults, including one or more of a conclusion (e.g., positive ornegative for a condition), actual numerical results of a test; the imageitself, and/or additional information pertinent to the medical record.

In accordance with one aspect of the invention, a non-transitorycomputer readable medium for updating an electronic medical record basedon patient-captured image data may contain instructions that whenexecuted by at least one processor cause the at least one processor toperform a method in accordance with aspects set forth in accordance withFIGS. 21-23 and discussed above.

The method may include accessing a record of a verification codeassociated with a home testing kit sent to a patient. A uniqueverification code may be associated with each individual home testingkit, or a plurality of testing kits. The verification code may berecorded in any manner such as in memory or as a hard copy. Memory mayinclude digital or analog memory. Memory may include Random AccessMemory (RAM) devices, NOR or NAND flash memory devices, Read Only Memory(ROM) devices, etc. The verification code may be accessed manually, orover any wired or wireless network or communication channel. Forinstance, accessing may include connecting to a remote server, or mayinclude connection from a mobile communications device over a cellularnetwork.

The method may include receiving a captured medical image of the patientvia a wireless network. The medical image of the patient may include afeature or characteristic of the patient, or may include an image of atest kit or portion thereof having been prepared by the patient. Afeature or characteristic may include a legion, mole, freckle, scar,injury, burn or other feature or characteristic of a body surface.

The method may include analyzing the medical image to determine aninsufficiency of the image for medical examination. The analysis may beperformed locally or remotely. For instance, if a patient captures amedical image on a mobile communications device, the mobilecommunications device may determine if the captured image is sufficient.Additionally, the analysis may be performed remotely if the medicalimage is transferred to a remote location. The determination ofinsufficiency may include an analysis of pixels, color, light, whetheror not an image includes a complete subject, or any other determinationthat may indicate an image is unsuitable.

The method may include sending at least one message indicative of theinsufficiency of the image, wherein the at least one message includes aguidance to capture a new image. The at least one message may include atext or SMS message, an email, a video, or any other visual display.Additionally, the at least one message may include an audible prompt orsound bite. The at least one message may include a plurality ofmessages, or a series of messages to aid a patient in capturing a newimage.

The method may include wirelessly receiving the new image from thepatient along with the verification code. Wirelessly receiving mayinvolve communications over any wireless network or system, such asWIFI, cellular, radio frequency, etc. The verification code may bereceived from the patient as an SMS or text message, numerical input,image, audio call, email, or via any other type of communication.

The method may include using the verification code to verify that thenew image was received from the patient. Upon verification, the methodmay include automatically sending a notification to a healthcareprovider for updating an electronic medical record of the patient withdata associated with the new image. The notification may be sent via anautomated messaging system, via a phone call, an SMS or text message, anemail, or any other method or system to provide notification to ahealthcare provider. The notification may include a test result orresults associated with a medical test performed by the patient, imagesor image-data from the medical test, a notification that a test wasperformed, or any other prompt or message which may be sent to ahealthcare provider to indicate an update to an electronic medicalrecord may be performed.

In accordance with one aspect of the invention, a system for updating anelectronic medical record based on patient-captured image data isdisclosed. The system may be configured to perform methods consistentwith aspects set forth in accordance with FIGS. 21-23 and discussedabove. The system may include at least one processor. The at least oneprocessor may be any of the devices or combinations of devices describedherein. Those devices may be embodied within one or more of a personalcomputer, laptop computer, desktop computer, tablet computer, notebooks,mobile phone, a terminal, a kiosk, PDA, a cloud-based computing device,a remote server or servers, smart phone, smart watch, smart device, orany other system allowing for processing of information.

In some embodiments, the at least one processor may be located remotefrom a mobile communications device of a patient and performfunctionality based on data received from the mobile communicationsdevice of the patient. For example, the processor may be configured toaccess a record of a verification code associated with a home testingkit sent to a patient. A unique verification code may be associated witheach individual home testing kit, or a plurality of home testing kits.For example, each testing kit may have its own unique code that may beassigned to a particular patient. Alternatively, a group of test kitsmay share a code that may be assigned to a group of individualsreceiving a common form of test kit.

The processor may be configured to receive a captured medical image ofthe patient via a wireless network, as previously discussed. The medicalimage of the patient may include a feature or characteristic of thepatient, or may include an image of a test kit or a portion thereofhaving been prepared by the patient. A feature or characteristic mayinclude a legion, mole, freckle, scar, injury, burn or other feature orcharacteristic of a body surface.

The processor may be configured to analyze the medical image todetermine an insufficiency of the image for medical examination.Although the analysis may be performed remote from the mobilecommunications device that sent the medical image, some or all of theanalysis may be performed locally on the mobile communications device.For instance, if a patient captures a medical image on a mobilecommunications device, the mobile communications device may determine ifthe captured image is sufficient, or at least meets a minimum threshold.Further analysis on a remote server may establish that the image isnevertheless insufficient. The determination of insufficiency may bebased on analysis of pixels, color, light, whether or not an imageincludes a complete subject, or any other determination that mayindicate an image is unsuitable.

The processor may be configured to send at least one message indicativeof the insufficiency of the image, wherein the at least one messageincludes a guidance to capture a new image. For example, if the messageis sent to the mobile communications device, the patient or agent of thepatient may use the mobile communications device to obtain and transfera new image to the remote processor. The at least one message mayinclude a text or SMS message, an email, a video, or any other visualdisplay. Additionally, the at least one message may include an audibleprompt or sound bite. The at least one message may include a pluralityof messages, or a series of messages to aid a patient in capturing a newimage.

The processor may be configured to wirelessly receive the new image fromthe patient as well as the verification code. Wirelessly receiving mayinvolve communications over any wireless network or system, such asWIFI, cellular, radio frequency, or any other communication channel asdiscussed above.

The processor may also be configured to use the verification code toverify that the new image was received from the patient; and uponverification, automatically send a notification to a healthcare providerfor updating an electronic medical record of the patient with dataassociated with the new image, as discussed previously.

Aspects of this disclosure may relate to medical testing, includingmethods, systems, devices, and computer readable media. For ease ofdiscussion, a method is described below, with the understanding thataspects of the method apply equally to systems, devices, and computerreadable media. For example, some aspects of such a method may occurelectronically over a network that is either wired, wireless, or both.Other aspects of such a method may occur using non-electronic means. Ina broadest sense, the method is not limited to particular physicaland/or electronic instrumentalities, but rather may be accomplishedusing many differing instrumentalities.

Consistent with disclosed embodiments, a method may involve receivingfrom a healthcare provider information identifying a plurality ofindividuals associated with a first insurance status. A healthcareprovider may include any physician, doctor, nurse, surgeon, agent of aphysician, hospital, clinic, insurance company, public healthconsultant, or any individual, group, organization or entity having aninterest in healthcare or providing health services. A plurality ofindividuals may include individuals of the general population, a subsetof the population, individuals having a pre-diagnosed or pre-existingcondition, or any person or animal that may be in need of a medical testor procedure. Insurance status may include any status based on adiagnosis, an insurance claim, benefits limits, cost or price ofcoverage, provider information or any status related to health orhealthcare. Any information related to health or healthcare may beutilized in determining or identifying an insurance status. For example,insurance coverage limits may differ between the first insurance statusand the second insurance status.

Aspects of this disclosure may involve delivering home testing kits tothe plurality of individuals. Some examples of such delivery aredescribed above in relation to method 2300. The testing kits, or testkit, may include any number of materials configured to determine acharacteristic or feature of an individual. A home testing kit mayinclude elements to enable monitoring, measuring, analyzing, orcapturing of information reflective of one or more characteristics orfeatures of a patient. For example, the test kits may allow for thedetermination of one or more of an analyte, compound, component,composition or property of or found within a biological fluid. The hometesting kits may include one or more cups, receptacles or padsconfigured to hold or extract one or more biological fluids. The kitsmay include one or more reagents configured to react with one or moreanalytes, compounds, components, compositions, features or chemicalsfound within a biological fluid. The reagents may be carried or embodiedby one or more of a dipstick, a pad, a sheet, a receptacle, or any meansto support or hold a reagent. One additional example of such kit mayinclude urinalysis kit 1200.

Each home testing kit may include a colorized surface including aplurality of colored reference elements. The colorized surface mayinclude one or more colors, hues, shades, shapes, indicia or featuresthat may be used for providing a reference during the use of a test kit.The colorized surface may take any form or shape, and may be made of anymaterial capable of supporting at least one colored reference element,or being printed on to include at least one colored reference element.The colorized surface may also be impregnated with one or more colors,hues, shades, shapes, indicia or features. The reference elements may beapplied, impregnated, printed, attached or affixed in any manner.

The term “home testing kit,” is intended to be interpreted broadly as akit that may be used in the home, but also a kit that could be used inother locations such as clinics, doctors' offices, nursing homes,convalescent centers, and other locations where a patient might visit orbe located.

Some aspects of this disclosure may involve receiving electronicallyfrom mobile communications devices of at least some of the plurality ofindividuals, medical image information corresponding to a medicalanalysis region in proximity to the colorized surface. Mobilecommunications devices may include any device with communicationscapabilities that enables transmission of image-related information. Themobile communications device may be enabled to transmit a medical imageor data related to a medical image. A device may be enabled to capture amedical image if it includes an integrated image sensor, if it has thecapability to pair with a device that includes an image sensor, or if ithas the ability to otherwise receive medical image information fromanother source for transmission. For example, a mobile phone or tabletmay integrate an image sensor with a transmission component.Alternatively, a non-networked camera paired with a PC or othernetworked device may serve as a mobile communications device. Theelectronic transmission may take place over any wired or wirelesscommunication channel and may be performed by any method allowing formoving data or information from one location to another. Thetransmission may even be performed by the transferring of data with aremovable memory from one device to another. A medical image may be anyimage used for medical purposes. For example, such a medical image mayinclude an image of a reaction to a particular test, an image of abiological response, an anatomical image, or an image of a wound or acondition. Medical image information may include an image or otherimage-related data, metadata, or information obtainable from a medicalimage. The medical image information may reflect a reaction or changeassociated any analyte, characteristic, composition, component, feature,parameter, etc. of any biological fluid. Additionally, the medical imageinformation may reflect information about a skin or body feature, suchas a size, shape, color, healing progress, infection, benign, cancerous,pre-cancerous, etc. or any other information obtainable from data,metadata, or information related to an image. A medical analysis regionmay include any portion of a home testing kit, any portion of a patient,or any other element that may allow for a determination or diagnosis.For example, a medical analysis region may include one or more of adipstick, a portion of the dipstick including one or more test reagentpads, a skin feature, or skin surface. A medical analysis region may bein proximity to the colorized surface during a portion of the testingprocedure. In proximity includes being, near, adjacent, next to, closeto, or otherwise being capable of being captured in a medical image viaan image sensor. A portion of the test kit may be placed on or aroundthe colorized surface prior to collection of a medical image.Alternatively, a colorized surface may be placed on or around a medicalanalysis region, such as a skin feature, prior to collection of amedical image.

In another aspect of the invention, the received medical imageinformation may be processed to determine a state of each correspondingmedical analysis region. The processing may include any data orinformation manipulation or analysis that allows for a determination tobe made. The processing may take place locally, such as on mobilecommunications device, or remotely on one or more servers. It is notedthat the processing can be performed by any processor having beenconfigured or programmed to perform such an operation. Determining astate of each corresponding medical analysis region may include anydetermination, analysis, or characterization capable of being obtainedfrom medical image information. For example, analyzing colors, hues,shades, or other visual characteristic of a plurality of reagent padsmay allow for a determination of one or more concentrations, levels, orpresence of different analytes, characteristics or components foundwithin a biological fluid. In another aspect, determining a state ofeach corresponding medical analysis region may include defining a skinfeature as benign, precancerous, cancerous, infected, scar, healing, orany determination that can be associated with a skin feature. Anotheraspect of the method may involve, based on the processed medical imageinformation, electronically identifying a group of individuals withmedical analysis regions in a differing state of criticality than othersof the plurality of individuals. A group of individuals may include oneor more individuals. In some examples, the group of individuals may beelectronically identified based on results of any one of process 700,process 800, process 1400, process 1700, process 2000, process 2300, andso forth.

Electronically identifying may include any data or informationprocessing, analyzing, or manipulation that can be performedelectronically. This may include any digital or analog analysisperformed with software, firmware, or hardware, and may also include anytransmission or manipulation of a file, data, information, notification,message, or alert over any wired or wireless communication network. Adiffering state of criticality may be determined by any quantifiable ormeasurable characteristic or feature from the received medical imageinformation. For example, a determination of the presence or absence ofan analyte in a sample may lead to an identification of a differingstate of criticality, the presence or absence of a skin feature may leadto an identification of a differing state of criticality, and theindication of compliance, no-compliance, or proper use of the hometesting kit may lead to a differing state of criticality. In anotheraspect, the method may include electronically providing the healthcareprovider with information indicating that there is a likelihood that thegroup of individuals is entitled to a second insurance status differentfrom the first insurance status.

Electronically providing may include the transmission of any message,alert, file, data or information over any wired or wirelesscommunication channel. A second insurance status may differ from thefirst insurance status in any manner. For example, differing insurancestatuses may be based on a degree or level of health, risk, treatmentcost, severity of a condition, one or more of the foregoing incombination with other patient specific information such as familymedical history, age, preexisting condition, geographical or situationalrisk factors, or any other information related to the patient. By way ofexample only, the first insurance status may be associated withindividuals who surpass a threshold, and a second insurance status maybe associated with individuals who fall below threshold. If test resultsindicate an elevated risk of kidney failure, for example, such resultsmay trigger a change in insurance status. Test results may cause eithera reduction or an increase in insurance status. For example, anindividual with a preexisting condition or risk may have an insurancestatus upgraded if test results suggest a marked improvement incondition or risk level. Alternatively, individuals who show a markeddecrease in condition or risk may have their insurance statusdowngraded. Thus, for example, a first status may indicate a healthyindividual, while a second status may be indicative of kidney failure oran increased likelihood of developing kidney failure, and may suggestthe individuals entitled to the second insurance status may need moretreatment, and thus incur additional healthcare expenses.

While the features previously described are not limited to a particularstructure or additional steps, by way of example only, FIG. 24illustrates one example of a medical testing method that may aidhealthcare providers to determine when conditions of a patient or agroup of patients deteriorate. FIG. 24 illustrates a plurality ofindividuals 2401. The plurality of individuals 2401 may be a portion ofthe general population. The plurality of individuals 2401 may or may nothave a pre-existing medical condition. In transmission 2410, theplurality of individuals each receive a home testing kit 2400, which mayinclude a dip stick 2402, colorized surface 2403, and container 2404,among potentially other items. Container 2404 may have an adjustablesize configured to contain a biological fluid. The dipstick 2402 maycontain a plurality of reagent pads thereon for measuring differingproperties of the biological fluid; and the colorized surface 2403 maycontain a dipstick placement region and a plurality of colored referenceelements greater in number than the plurality of reagent pads. Thecontainer 2404 may be adjustable in size from a smaller volume to alarger volume, or may be collapsible. This may be achieved throughtelescoping, bellows, or another feature. The biological fluid may beurine, sweat, stool, blood, breast milk, interstitial fluid, saliva orany other biological fluid or sample. The plurality of reagent pads mayinclude any number greater than one, and may be configured to react withone or more characteristics, analytes, compounds, compositions,properties, features, etc. of the biological fluid. Moreover, onereagent pad may be configured to reflect a characteristic of a firstbiological fluid, and a second reagent pad may be configured to reflecta characteristic of a second biological fluid. The colorized surface2403 may contain a plurality of colored reference elements greater innumber than a plurality of reagent pads included on the dipstick 2402.Dipstick placement region may include a recess, a delineated section, amark, or any feature that may direct a user as to where to place thedipstick.

The test kit may also include instructions for use (or a link toinstructions for use). Regardless of the precise contents of the testkit and their intended use, the user might be instructed on how toperform a test, capture an image of an area of interest (e.g., at teststrip) adjacent the colorized surface 2403, and to capture an associatedimage using mobile communications device 2405, such as a mobile phone.Some examples of mobile communications device 2405 may include any oneof mobile communications device 115, mobile communications device 1604a, mobile communications device 1803, mobile communications device 2200,a smartphone, a tablet, a wearable computer, and so forth.

In the image, the colorized surface may be used to calibrate for locallighting conditions in order to determine rectified reference colors ofthe medical analysis region. For example, colorized surface 2403 mayinclude a plurality of colored reference elements or other featureswhich allow for calibration of local lighting conditions. This allowsfor analysis irrespective of local illumination conditions. For example,the method may allow for proper analysis in an overly bright room fullof natural light, as well as a dim-lit room having fluorescent lighting.The home testing kits may further include a blot pad (not illustrated)for removing excess biological fluid from the dipstick after beingdipped in the biological fluid, to thereby enable non-distorted imagecapture of the plurality of reagent pads by an image sensor. The blotpad may be formed of any suitable material for removing excess fluidsuch as cloth, paper, natural or synthetic fibers, porous material, orany other material having an absorbent, hydrophilic or wicking quality.An image sensor may include any sensor capable of detecting andconverting optical signals in the near-infrared, infrared, visible, andultraviolet spectrums into electrical signals.

Medical image information of at least some of the plurality ofindividuals 2401 may be transmitted via communication channel 2411 frommobile communications device 2405 to processing device 2406. Although asingle test kit 2400, single mobile communications device 2405 andsingle communication channel 2411 are illustrated in FIG. 24, it is tobe understood that each of the plurality of individuals 2401 will use aseparate test kit and will likely capture the medical image on separatemobile communications devices for transmission to processing device 2406via separate channels. Thus, the singular illustrations in FIG. 24 arefor ease of discussion only and are intended to encompass the plural.Mobile communications devices 2405 may include one or more of a phone,mobile phone, smart phone, smart watch, tablet, laptop computer,personal computer, PDA, or any device which may allow for communicationbetween a patient and the system. Mobile communications devices mayinclude one or more image sensors or may receive information from one ormore image sensors. The medical image information may include an imageor other image-related data, metadata, or information. In one aspect,the medical image information may reflect a resulting color of achemical reaction between a biological fluid and the at least onereagent test pad. The color may be any color, and the chemical reactionmay take place with any desired analyte, characteristic, composition,component, feature, parameter, etc. of or in the biological fluid. Whenthe biological fluid is urine, the medical image information may reflectan albumin to creatine ratio. The medical image information may reflecta reaction or change associated any analyte, characteristic,composition, component, feature, parameter, etc. of any biologicalfluid. In another aspect, the medical analysis region includes a skinfeature, and the medical image information includes at least one of animage of the skin feature adjacent the colorized surface or data derivedfrom the image of the skin feature adjacent the colorized surface. Theskin feature may be one or more of a freckle, mole, cut, scrape, injury,mark, boil, legion, scar, etc. or any feature present on a surface ofthe skin, or underlying the surface of the skin. The medical imageinformation may reflect information about the skin feature, such as asize, shape, color, healing progress, infection, benign, cancerous,pre-cancerous, etc. or any other information obtainable from data,metadata, or information related to an image.

The medical analysis region may include a portion of the dipstick 2402,including one or more of the reagent pads, a feature of a skin or bodysurface, or any other feature or element that may be used in conjunctionwith colorized surface 2403 to perform an analysis. The medical analysisregion may include at least one reagent test pad, and the medical imageinformation may include at least one of an image of the at least onereagent test pad adjacent the colorized surface or data derived from theimage of the at least one reagent test pad adjacent the colorizedsurface. The at least one regent test pad may be one of a plurality ofreagent test pads configured to react with one or more of an analyte,component, characteristic, property, feature, etc. within or associatedwith a biological fluid. Adjacent the colorized surface may includebeing positioned on, in, near, next to, or proximate a colorizedsurface.

A server or other processing device 2406 may process the receivedmedical image information to determine a state of each correspondingmedical analysis region. The server or other processing device 2406 maybe a remote server or servers, a cloud-based server or servers, a localprocessing device such as a computer, CPU, microprocessor, or may beintegrated into a mobile communications device 2405. Determining a stateof each corresponding medical analysis region may include anydetermination, analysis, or characterization capable of being obtainedfrom medical image information. For example, analyzing colors, hues,shades, etc. of a plurality of reagent pads may allow for adetermination of one or more concentrations, levels, or presence ofdifferent analytes, characteristics or components found within abiological fluid. In another aspect, determining a state of eachcorresponding medical analysis region may include defining a skinfeature as benign, precancerous, cancerous, infected, scar, etc. Forexample, the skin feature may be a mole and a determined state of theskin feature may be indicative of an increased likelihood that the moleis cancerous. Additionally, the skin feature may be a wound and adetermined state of the skin feature is indicative of wound healingprogress. Such a determinization may be made over a time utilizing aplurality of, or series of images. Moreover, such a determination may bemade by comparison to one or more benchmarks or thresholds. The one ormore benchmarks or thresholds may be based on research, medicalliterature, surveys, tests, etc. or based on a specific patient's priormedical image.

A method, system, or computer readable media of this disclosure mayinclude, updating personal electronic medical records of the pluralityof individuals with test results associated with the received medicalinformation. In some examples, the received medical information may bebased on process 700 and/or process 800 and/or process 1400 and/orprocess 1700 and/or process 2000 and/or process 2300. The plurality ofindividuals may include one or more of the plurality of individuals2401. Test results may include concentrations, levels, characteristics,features, parameters, etc. of one or more components or analytes foundwithin a biological fluid. Test results may also include raw data orother information associated with performing a home test. For example,original transmitted images may be included as test results as well asconcentrations of specific compounds, or colors reflecting thoseconcentrations.

Based on the processed medical image information, server or otherprocessing device 2406 may electronically identify a group ofindividuals with medical analysis regions in a differing state ofcriticality than others of the plurality of individuals 2401. Asillustrated in FIG. 24, a first group of individuals 2407 may bedetermined to have a medical analysis region in a differing state ofcriticality than a second group of individuals 2408. For example, amachine learning model may be trained using training examples todetermine states of criticality of individuals from medical imageinformation, and the trained machine learning model may be used toanalyze the medical image information associated with an individual anddetermine the state of criticality of the individual. An example of suchtraining examples may include medical image information together with alabel indicative of a state of criticality.

A differing state of criticality may be determined by any quantifiableor measurable characteristic or feature from the received medical imageinformation. For example, medical image information indicating thepresence of blood in urine samples of a portion of the population 2401may be the basis for identifying a group of individuals with medicalanalysis regions in a differing state of criticality. Additionally, thepresence or absence of a skin feature may lead to an identification of adiffering state of criticality. Also, the differing state of criticalitymay indicate compliance, non-compliance, or proper use of the hometesting kit.

A method, system, or computer readable media associated with thisdisclosure may include storing data associated with past medical imageinformation of the plurality of individuals associated with a firstinsurance status, and identifying the state of criticality by comparingcurrent medical image information with past medical image information.Data may be stored in one or more databases locally or remotely, onservers or a plurality of servers, on one or more cloud-based devices,ROM, RAM, or any storage device that may allow for querying or analysisof stored data. Medical image information may include one or moreimages, metadata associated with one or more images, calculations,results, or other information associated with or obtainable from one ormore images, or any other information related with one or more medicalimages.

As illustrated in FIG. 24, the second group of individuals 2408 may besubject to additional testing at a later time (step 2414). Theseindividuals 2408 may overlap with a subgroup of individual from group2401. Periodically delivering home testing kits 2400′ to some of thealready tested individuals in group 2401 may enable those who werepreviously not entitled to an insurance status change (or who wereoriginally subject to a change) to obtain a later revision of insurancestatus as the result of updated testing. This periodic retesting may beinitiated by delivering home testing kits at a predetermined timeperiod, such as annually or bi-annually.

Systems, methods, and computer readable media of this disclosure mayinclude electronically providing the healthcare provider withinformation indicating that there is a likelihood that the group ofindividuals 2407 is entitled to a second insurance status different fromthe first insurance status. Providing the healthcare provider with theinformation that there is a likelihood that the group of individuals2407 is entitled to a second insurance status different from the firstinsurance status may cause an automatic update of the insurance statusof the group of individuals 2407. The group of individuals 2407 mayinclude any number of individuals, from a few to a hundred or even manymore individuals. The number of individuals in a group may be definedover a predetermined period of time. For example, the number ofindividuals in a group may increase up to a period of one year or more.In some examples, a machine learning model may be trained using trainingexamples to determine whether individuals are entitled to a particularinsurance status from states of criticality of individuals, and thetrained machine learning model may be used to analyze a state ofcriticality of an individual to determine whether the individual isentitled to the particular insurance status (or a likelihood that theindividual is entitled to the particular insurance status). An exampleof such training example may include a state of criticality of anindividual together with a label indicating whether the individual isentitled to a particular insurance status (or a likelihood that theindividual is entitled to the particular insurance status).

An automatic update of the insurance status may be performed over anywired or wireless communication pathway. An automatic update may alertone or more end users of a change in insurance status of the group ofindividuals. As illustrated in FIG. 24, a first group of individuals2407 may be entitled to an insurance status different from an insurancestatus of the second group of individuals 2408, or different from aninitial insurance status. Insurance status may include any status basedon a diagnosis, an insurance claim, benefits limits, cost or price ofcoverage, provider information or any status related to health orhealthcare. Coverage limits may differ between the first insurancestatus and the second insurance status. For example, if it is determinedthat the first group of individuals 2407 have received resultsindicating kidney failure or an increased likelihood of developingkidney failure, the second insurance status may suggest the first groupof individuals 2407 may need more treatment, and thus incur additionalhealthcare expenses.

FIG. 25 illustrates an embodiment for increasing compliance among theplurality of individuals receiving a home testing kit. This embodimentmay include identifying a first sub-plurality of individuals whotransmitted medical image information within a time period; identifyinga second sub-plurality of individuals who did not transmit medical imageinformation within the time period; analyzing metadata of the medicalimage information associated with the first sub-plurality of individualsto determine at least one pattern for completing a medical procedureassociated with the home testing kits; and sending a reminder to atleast one member from the second sub-plurality of individuals based onthe determined at least one pattern. As shown in step 2501, home testingkits may be delivered to a plurality of individuals. Step 2502 includesdetermining if an individual having received a home testing kit sendsmedical image information within a predetermined time period. A firstsub-plurality of individuals who transmitted medical image informationwithin a time period may be identified at step 2503. A secondsub-plurality of individuals who did not transmit medical imageinformation within the time period may be identified at step 2506.Analyzing metadata of the medical image information associated with thefirst sub-plurality of individuals may be performed at step 2504. Themetadata of the medical image information may include times when themedical image information was transmitted, locations from which themedical information was transmitted, and information characterizingindividuals such as individuals from the first sub-plurality ofindividuals. The metadata of the medical image information mayadditionally include time stamps, weather information, othergeographical information, image sensor information, etc. or any otherinformation capable of being obtained from, inferred, or associated withan image. Characterizing information may include age, gender, workplace,income, socio-economic status, race, weight, lifestyle information, etc.

The metadata may include any data or information that may be useful todetermine at least one pattern for completing a medical procedureassociated with the home testing kit as in step 2505. Patterns may beassociated with a day of the week, time of day, time of the month,season, weather, where an individual lives, the age of an individual,work place, or any characteristic that may be analyzed to identify atleast one pattern among individuals that complete a medical test. Forexample, a pattern may associate a subgroup of individuals withparticular characteristic that increases and/or decrease the likelihoodfor completing the medical procedure. In one example, a RANdom SAmpleConsensus (RANSAC) algorithm may be used to identify the subgroup ofindividuals and the particular characteristic. Upon determining acompliance pattern, the pattern may be used for determining remindercharacteristics for use with the second sub-plurality of individuals atstep 2507, for example by selecting reminder characteristics thatincreases the likelihood of individuals in the second sub-plurality ofindividuals to complete the medical procedure according to thedetermined at least one pattern and/or by avoiding remindercharacteristics that decreases the likelihood of individuals in thesecond sub-plurality of individuals to complete the medical procedureaccording to the determined at least one pattern. A reminder may then besent to at least one member from the second sub-plurality of individualsat step 2508 based on the determined at least one pattern andcharacteristics of the pattern (for example, based on the determinedreminder characteristics). The at least one pattern may include one ormore patterns. In one aspect, the determined at least one pattern may bean indication that adherent patients tended to comply on a particularweekday, and wherein sending the reminder based on the determined atleast one pattern includes timing the reminder to coincide with theparticular weekday. For example, in response to a determination thatsome individuals tend to comply on Fridays, a reminder may be sent toone or more of these individuals on a Friday. Additionally, if thepattern indicates that individuals tend to comply during the evening onFridays, the reminder may be sent to coincide with a Friday evening. Inanother aspect, the determined at least one pattern may be an indicationthat adherent patients tended to comply when presented with a particularmessage, and wherein sending the reminder based on the determined atleast one pattern includes sending the reminder with the particularmessage. The message may be one or more of a text or SMS message, anemail, a letter, a song, a sound bite or clip, an animation, a prompt, avibration or any other audio, visual, or tactile mode of communication.

FIG. 26 is a flow chart illustrating a medical testing method 2600 inaccordance with the disclosure set forth above. A non-transitorycomputer readable medium may contain instructions that when executed bya processor cause the processor to perform the medical testing method2600. The method may include receiving from a healthcare providerinformation identifying a plurality of individuals associated with afirst insurance status. Step 2601 illustrates identifying individualswith a first insurance status. A healthcare provider may include anyindividual, organization or group of individuals or organizations, asdiscussed previously. A first insurance status may include any statusbased on a diagnosis, an insurance claim, benefits limits, cost or priceof coverage, provider information or any status related to health orhealthcare.

The method may include generating a list of a plurality of individualsto whom home testing kits are to be sent, wherein each home testing kitincludes a colorized surface including a plurality of colored referenceelements. A list of a plurality of individuals to whom home testing kitshave been or are to be sent may be generated and test kits may bedelivered to the individuals at step 2602. The plurality of individualswhom receive test kits may include one or more of the individuals havinga first insurance status. A list may be generated or populatedautomatically, manually, or from querying a database, memory, orclearinghouse.

The method may include receiving electronically from mobilecommunications devices of at least some of the plurality of individuals,medical image information corresponding to a medical analysis region inproximity to the colorized surface. Step 2603 illustrates receivingimage information corresponding to the analysis region. The medicalimage information may be sent or transmitted over any wired or wirelesscommunication channel, over communication network 150, and so forth. Themedical information may include an image, a portion of an image,metadata associated with an image or a portion of an image, or any otherinformation or data associated with the medical image. The medicalanalysis region may include a dipstick, a portion of the dipstickincluding one or more test reagent pads, a skin feature, skin surface,etc.

The method may further include processing the received medical imageinformation to determine a state of each corresponding medical analysisregion. A state of each corresponding medical analysis region may bedetermined at step 2604. A state of each corresponding medical analysisregion may include detecting or analyzing a color, characteristic,property, feature, etc. of an analysis region. The state of eachcorresponding medical analysis region may additionally include adiagnosis, or may indicate compliance with a medical test. For example,a state of a test reagent may include the diagnosis of a concentrationof a specific analyte found in a biological fluid or other sample, ormay merely indicate that a test has been performed, indicating patientcompliance. For example, the processing of the received medical imageinformation may include any of the analysis techniques described above.

The method may include based on the processed medical images,electronically identifying a group of individuals with medical analysisregions in a differing state of criticality than others of the pluralityof individuals. At step 2605, individuals having analysis regions indifferent states may be identified. Electronically identifying mayinclude any analysis, computation, comparison, etc. capable of beingperformed by a processor or being transmitted over any wired or wirelesscommunication channel. A group of individuals may include one or moreindividuals. A differing state of criticality includes any detectable ormeasurable quality or feature of an analysis region.

The method may additionally include electronically providing thehealthcare provider with information indicating that there is alikelihood that the group of individuals is entitled to a secondinsurance status different from the first insurance status.Electronically providing may include the transmission of a message,alert, file, data, information, etc. over any wired or wirelesscommunication channel. A second insurance status may differ from thefirst insurance status in any manner. For example, the first insurancestatus may indicate a healthy individual, while the second status may beindicative of an unhealthy individual, such as, for example, someone atrisk of or exhibiting symptoms of kidney failure, and may suggest theindividuals entitled to the second insurance status may need moretreatment, and thus incur additional healthcare expenses.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, e.g., hard disks or CD ROM, or otherforms of RAM or ROM, USB media, DVD, Blu-ray, Ultra HD Blu-ray, or otheroptical drive media.

Computer programs based on the written description and disclosed methodsare within the skills of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. The examplesare to be construed as non-exclusive. Furthermore, the steps of thedisclosed methods may be modified in any manner, including by reorderingsteps and/or inserting or deleting steps. It is intended, therefore,that the specification and examples be considered as illustrative only.

1. A color board for use in reagent strip testing, the color boardcomprising: a color board surface; a first colored reference elementprinted on the color board surface; a second colored reference elementprinted on the color board surface; a test region on the color boardsurface configured to receive at least one reagent pad; and a uniquecode on the color board, the code reflecting specific chromaticproperties associated with each of the first colored element and thesecond colored element at the time of printing, and wherein the code ismachine readable to enable a machine to later normalize a comparisoncolor, for determining chromatic properties of the at least one reagentpad.
 2. The color board of claim 1, wherein the chromatic properties aredecipherable from the code.
 3. The color board of claim 1, wherein thechromatic properties are determinable by comparing the code withinformation stored in a data structure.
 4. The color board of claim 1,wherein the chromatic properties reflected by the code include at leastone of hue, value, and chroma.
 5. The color board of claim 1, whereinthe chromatic properties reflected by the code include variations incolors of the first colored reference element and the second coloredreference element.
 6. The color board of claim 1, wherein normalizing acomparison color for determining the chromatic properties of the atleast one reagent pad includes removing local illumination effects. 7.The color board of claim 1, wherein the code is configured to enable amachine to later determine at least one detail regarding a single hometesting kit associated with the color board, and to execute testing ofthe at least one reagent pad based on the at least one detail.
 8. Thecolor board of claim 1, wherein the code is configured to enable amachine to later determine if the color board was used more than apredefined number of times.
 9. The color board of claim 1, wherein thecode is configured to enable a machine to later determine if the colorboard was previously used.
 10. The color board of claim 1, wherein thefirst colored reference element and the second colored reference elementshare substantially a same color.
 11. The color board of claim 10,wherein the first colored reference element and the second coloredreference element are arranged on the color board on opposing sides ofthe test region.
 12. The color board of claim 1, wherein the test regionis configured to accommodate a reagent pad.
 13. The color board of claim1, wherein the test region is configured to accommodate a dipstick. 14.The color board of claim 1, wherein the test region is configured toaccommodate a wound.
 15. A computer program product containinginstructions that when executed by at least one processor, cause the atleast one processor to carry out a method for analyzing visible chemicalreactions, the method comprising: receiving from an image sensorassociated with a mobile communications device an image of a reagent padin proximity to a color board having a first colored reference, a secondcolored reference, a test region on the color board surface configuredto receive at least one reagent pad, and a unique code; determining fromthe unique code specific chromatic properties associated with each ofthe first colored element and the second colored element; and using thespecific chromatic properties of first colored reference element and thesecond colored reference element to analyze a depiction of the at leastone reagent pad in the image for determining an extent of a chemicalreaction on the at least one reagent pad.
 16. The computer programproduct of claim 15, wherein the method further includes: in response tofirst specific chromatic properties determined from the unique code anda first depiction of the at least one reagent pad in the image,determining a first extent of the chemical reaction on the at least onereagent pad; and in response to second specific chromatic propertiesdetermined from the unique code and the first depiction of the at leastone reagent pad in the image, determining a second extent of thechemical reaction on the at least one reagent pad, the second extentdiffers from the first extent.
 17. The computer program product of claim15, wherein the method further includes: using the unique code to selecta type of medical information to be provided; and based on thedetermined extent of the chemical reaction on the at least one reagentpad, providing medical information of the selected type of medicalinformation.
 18. The computer program product of claim 15, wherein themethod further includes causing the mobile communications device toprovide data representing the extent of the chemical reaction.
 19. Thecomputer program product of claim 15, wherein the method furtherincludes: using the unique code to determine whether a number of usagesof the color board exceeds a selected threshold; in response to adetermination that the number of usages of the color board do not exceedthe selected threshold, providing first information based on thedetermined extent of the chemical reaction on the at least one reagentpad; and in response to a determination that the number of usages of thecolor board exceeds the selected threshold, forgoing providing firstinformation.
 20. The computer program product of claim 15, wherein themethod further includes: using the unique code to select a billingaccount; and causing an update to the selected billing account.